docs(vault): one page per endpoint, forever -- 45 era pages fold into three

Every payload swap a live endpoint has ever run is now one row in that
endpoint's Eras table, with its rollback path in the same row. No per-attempt
page, no per-swap ledger, nowhere else.

- Added: LLM/frank-archon.md (radiance, 2x R9700 TP=2), LLM/frank-dragoon.md
  (halogen flash-server, 19 eras), LLM/frank-zealot.md (llama.cpp PrismML fork
  on the 3090). Each carries Current shape, Eras, Laws, wire contract, files.
- Deleted: 45 per-payload runbooks -- the frank-dragoon line (11 including the
  pre-name halogen/kingjones/kairic/CIRU/eemin/agentionai/dflash2 pages and the
  benchmark page), the frank-zealot line (llama.cpp/vLLM/ik_llama/SGLang eras),
  the frank-archon line (vLLM/FreeToken/R9V/llama.cpp/radiance eras), plus
  frank-medium, frank-npu and the ubatch/PCIe one-offs that owned nothing.
- Rewired: every wikilink and markdown-path reference to a folded page points
  at the endpoint page instead; the three Research see-also lists and the
  ubatch sweep's runbook link included. No anchored links existed in the set.
- Regenerated both index pages as pure locators -- one row, one clause, no
  numbers -- so the docs_completeness locator law holds for all 153 pages.

tests/docs_completeness.bats green: index links every page, no unresolved
links. The pages moved to .AGENTTRASH/20261018-doc-dedupe/LLM/ (gitignored,
and git history holds every byte).

serverAppConfigs-docs/LLM/DFlash DSpark Speculative Decoding.md

@@ -250,7 +250,7 @@ docker run --rm --name vllm-qwen36-dflash \
 
 ## Deployment on Dankclaw (AMD Strix Halo gfx1151)
 
-`frank-dragoon` on Dankclaw ran the llama.cpp pipeline-parallel head (Qwen3.8-27B-Uncensored Q6_K) on port 7770, RPC-split with the Dankserver worker. Spec was the standalone MTP draft over the ngram-mod fallback, 1 slot at 262144 ctx, q8_0/f16 KV, head GPU side ROCm0. This page is a historical record of that era (mid-August 2026); for the current deployment see [[kingjones-strix-rocmfp4]].
+`frank-dragoon` on Dankclaw ran the llama.cpp pipeline-parallel head (Qwen3.8-27B-Uncensored Q6_K) on port 7770, RPC-split with the Dankserver worker. Spec was the standalone MTP draft over the ngram-mod fallback, 1 slot at 262144 ctx, q8_0/f16 KV, head GPU side ROCm0. This page is a historical record of that era (mid-August 2026); for the current deployment see [[frank-dragoon]].
 
 History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 2026-08-15 after a ROCm illegal-memory-access crash under two concurrent 16k prompts) and before that bundled MTP (`draft-mtp`, retired same day). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`), the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`), and the DSpark draft (`Qwen3.8-27B-DSpark-Q8_0.gguf`) remain in `/home/dank/models/` as reference only.
 
@@ -315,7 +315,7 @@ not switch without re-measuring".
 
 The A/B vehicle for ANY single-GPU llama.cpp knob (batch/ubatch, KV type,
 slot split, quant rung — not just spec flags) is the transient systemd
-drop-in procedure in [[frank-dragoon-benchmark]] and [[frank-zealot-benchmark]].
+drop-in procedure in [[frank-dragoon]] and [[frank-zealot]].
 
 ## Is it worth it? (arithmetic before deployment)
 
@@ -419,4 +419,4 @@ measurement that would settle it.
 - [[GGUF Compatibility]] — header parsing and custom tensor-type lock-in
 - [[Llama Cpp Serving Sizing]] — slot-pool and drafter KV sizing arithmetic
 - [[Llama Cpp Multi-Device Splits]] — where the drafter's KV actually lands
-- [[frank-dragoon-benchmark]] and [[frank-zealot-benchmark]] — the single-GPU env drop-in A/B vehicle
+- [[frank-dragoon]] and [[frank-zealot]] — the single-GPU env drop-in A/B vehicle

serverAppConfigs-docs/LLM/Dankclaw Improvements.md

@@ -18,8 +18,8 @@ marked "verify first". Re-verified 2026-08-15/16; corrections marked below.
 
 **Live today (2026-10-18).** The era described below — llama.cpp on the 8060S
 iGPU behind :7770 — is retired. Dankclaw now serves
-[[frank-dragoon-flash-server-v2-native]] (halogen-flash-server 0.16.2, port
+[[frank-dragoon]] (halogen-flash-server 0.16.2, port
-7770) and [[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML fork on
+7770) and [[frank-zealot]] (llama.cpp PrismML fork on
 the RTX 3090, port 7771 — drift today: that unit is `failed` since 2026-10-18
 03:40 and a hand-run 9B-MTP process answers the port), and boots `iommu=pt` since 2026-10-18
 ([[dankclaw-iommu-translation-off]]). The numbers below are the measurements of
@@ -32,7 +32,7 @@ re-checked 2026-10-18 and still hold.
   (b10433): Vulkan 25.13 t/s vs ROCm 24.33 t/s decode at 16k; at 262k ctx
   Vulkan decodes 48% faster (5.68 vs 3.83 t/s). ROCm wins prefill (410 vs
   302 t/s @64k). The "HIP out-performs Vulkan for decode" claim did not
-  hold on the 8060S. See [[frank-dragoon-benchmark]].
+  hold on the 8060S. See [[frank-dragoon]].
 - **Power profile**: the `/sys/class/drm/card0/device/power_profile` file
   does not exist on this machine (checked 2026-08-15). GPU DPM is already
   `performance`, forced level `auto`. This item is N/A.
@@ -67,7 +67,7 @@ The TTM row above is still live: `pages_limit = 29883536` (114 GiB) and
 2026-10-18).
 
 This split is the Strix Halo GTT/VGM design. It drives the batch/ubatch memory
-limits documented in [[frank-dragoon-benchmark]].
+limits documented in [[frank-dragoon]].
 
 ## 1. Self-Hosted Agentic Tools
 
@@ -130,7 +130,7 @@ All require root and a reboot for the modprobe change.
 
 ## Sources
 
-- [[frank-dragoon-benchmark]] — measured backend/spec numbers
+- [[frank-dragoon]] — measured backend/spec numbers
 - [[deployment]] — deploy and update process
 - AMD ROCm RDNA3.5 system optimization guide (gfx1151 memory/BIOS)
 - llama.cpp source: `ggml/CMakeLists.txt`, `ggml/src/ggml-hip/CMakeLists.txt`,
@@ -139,7 +139,7 @@ All require root and a reboot for the modprobe change.
 
 ## See Also
 
-- [[frank-dragoon-benchmark]]
+- [[frank-dragoon]]
 - [[DFlash DSpark Speculative Decoding]]
 - [[deployment]]
 - [[index]]

serverAppConfigs-docs/LLM/Dankclaw System Specs.md

@@ -10,7 +10,7 @@ timestamp: 2026-09-03
 
 ## Server Overview
 
-Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp inference through 2026-08-22 ([[ciru-activefpx-promptforge]], retired). The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same physical RAM pool. Today it serves [[frank-dragoon-flash-server-v2-native]] (halogen-flash-server 0.16.2 on the iGPU, port 7770) and [[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML fork on the RTX 3090, port 7771).
+Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp inference through 2026-08-22 ([[frank-dragoon]], retired). The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same physical RAM pool. Today it serves [[frank-dragoon]] (halogen-flash-server 0.16.2 on the iGPU, port 7770) and [[frank-zealot]] (llama.cpp PrismML fork on the RTX 3090, port 7771).
 ## Hardware Specifications
 
 ### CPU
@@ -46,7 +46,7 @@ Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp
 | Runtime | None installed. FastFlowLM 0.9.45 + XRT (installed 2026-08-16) was **retired 2026-09-11** with the `amd_iommu=off` decision — see [[dankclaw-iommu-translation-off]]; the package remains at `/home/dank/GitPackages/fastflowlm_0.9.45_ubuntu26.04_amd64.deb` |
 | Status | Not in service — no NPU endpoint is deployed. The device itself is usable since the 2026-10-18 `iommu=pt` boot (`/dev/accel/accel0` present, `crw-rw---- root:render 261,0`); Gitea issue #3 tracks enablement. See [[FastFlowLM NPU Inference]] |
 
-The NPU endpoint is retired (2026-09-11; [[frank-npu]]). The `frank-zealot` name now serves **Ternary-Bonsai-2-27B-PQ2_0** on the RTX 3090 via the llama.cpp PrismML fork (see [[frank-zealot-llama-prism-bonsai2-pq20]]); the Gemma4-12B QAT rung it once pointed at is retired history ([[frank-zealot-gemma4-12b]], 2026-09-17 → 10-01).
+The NPU endpoint is retired (2026-09-11; [[frank-npu]]). The `frank-zealot` name now serves **Ternary-Bonsai-2-27B-PQ2_0** on the RTX 3090 via the llama.cpp PrismML fork (see [[frank-zealot]]); the Gemma4-12B QAT rung it once pointed at is retired history ([[frank-zealot]], 2026-09-17 → 10-01).
 
 ### ROCm
 
@@ -95,21 +95,21 @@ cable between claw and server is required for 10G.
 | Kernel | 7.0.0-38-generic (`uname -r`, 2026-10-18) |
 | Boot line | `iommu=pt` since 2026-10-18 (`/proc/cmdline`, no `amd_iommu` token); eras before: `amd_iommu=off` 2026-09-11 → 10-18, IOMMU default-on before that — [[dankclaw-iommu-translation-off]] |
 | Python | 3.14.4 (system, `python3 --version`, 2026-10-18). The 3.12.13 CIRU venv left with that runtime (retired 2026-08-22) |
-| CIRU runtime | ciru-ai/ROCmFPX — **retired 2026-08-22** ([[ciru-activefpx-promptforge]]); not the serving runtime |
+| CIRU runtime | ciru-ai/ROCmFPX — **retired 2026-08-22** ([[frank-dragoon]]); not the serving runtime |
 | uv | 0.12.8 (`~/.local/bin/uv --version`, 2026-10-18) |
 
 ## Runtime Relevance
 
 - Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
 - The GPU budget is the GTT aperture: ~114 GiB (`mem_info_gtt_total` in `/sys/class/drm/card1/device/` = 122,402,963,456 B; `card1` is the amdgpu iGPU, `PCI_ID=1002:1586`; `llama-server --list-devices` reports 116,732 MiB). rocm-smi's smaller SVM/dedicated carve-out is **not** the budget — the 2026-08 "64 GiB rocm-smi" lesson is stale. Do not use `free` or `dmidecode` for the GPU budget.
-- The active services are `frank-dragoon` on 7770 — **halogen-flash-server 0.16.2** serving the native v2 abliterated `.hgn` in a rootless podman container ([[frank-dragoon-flash-server-v2-native]]) — and `frank-zealot` on 7771 ([[frank-zealot-llama-prism-bonsai2-pq20]]). The gufo v0.1.1 / UD-Q8_K_XL runtime this page once described is retired (2026-09-28 → 10-01, [[frank-dragoon-gufo-qwen38-27b]]); the gufo build tree is still on disk at `/home/dank/GitPackages/gufo-v0.1.1/`.
+- The active services are `frank-dragoon` on 7770 — **halogen-flash-server 0.16.2** serving the native v2 abliterated `.hgn` in a rootless podman container ([[frank-dragoon]]) — and `frank-zealot` on 7771 ([[frank-zealot]]). The gufo v0.1.1 / UD-Q8_K_XL runtime this page once described is retired (2026-09-28 → 10-01, [[frank-dragoon]]); the gufo build tree is still on disk at `/home/dank/GitPackages/gufo-v0.1.1/`.
 - The gufo-era note that the active runtime links against the host ROCm (`/opt/rocm` → `/etc/alternatives/rocm` → 7.2.4) through its RPATH belongs to that retired source build. The serving runtimes do not depend on the host ROCm that way: halogen runs inside its digest-pinned image, and `frank-zealot` is a CUDA build on the 3090. The retired CIRU venv's TheRock 7.15 SDK (`LD_LIBRARY_PATH`) was never in the serving path either.
 - The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18.
 - 424 GB free on `/` (`df -h`, 2026-10-18) is ample for model storage.
 
 ## See Also
 
-- [[ciru-activefpx-promptforge]]
+- [[frank-dragoon]]
 - [[Shared Memory Architecture Dankclaw]]
 - [[frank-redo-2026-08-17]]
 - [[FastFlowLM NPU Inference]]

serverAppConfigs-docs/LLM/Dankserver System Specs.md

@@ -16,7 +16,7 @@ timestamp: 2026-09-20
 ## Server Overview
 
 Dankserver is a media/library server and the `frank-archon` LLM host (radiance
-1.0.13, 2×R9700 tensor-parallel — [[frank-archon-radiance-qwen38-iq4r]]; the
+1.0.13, 2×R9700 tensor-parallel — [[frank-archon]]; the
 `frank-medium` name this page carried was retired 2026-09-20, see §Active
 Services). The host runs Ubuntu 26.04 (Resolute Raccoon). The RTX 3090 that
 served the FreeToken/CUDA `frank-zealot` era was removed on 2026-09-15 and
@@ -101,7 +101,7 @@ RCCL-backed rebuild needs no new host packages.
 | Kernel | 7.0.0-38-generic, `iommu=pt` (`uname -r`, `/proc/cmdline`, read 2026-10-18) |
 | Python | 3.14.4 |
 | ROCm userspace | Ubuntu 7.1 packages in /usr (hipcc 7.1.1, rocminfo, rocm-smi 7.1.1) |
-| Serving stack | radiance 1.0.13 container (`stilldeadcode/radiance@sha256:4c3f484564…`), TP=2 across both R9700s — [[frank-archon-radiance-qwen38-iq4r]]. The llama.cpp `fb27a525` build at `~/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714` is the retired frank-medium artifact |
+| Serving stack | radiance 1.0.13 container (`stilldeadcode/radiance@sha256:4c3f484564…`), TP=2 across both R9700s — [[frank-archon]]. The llama.cpp `fb27a525` build at `~/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714` is the retired frank-medium artifact |
 
 The stale AMD-repo ROCm 6.3.3 packages (167, `~24.04`-suffixed) that used to
 hijack `libhsa-runtime64` were purged on 2026-09-15 — see
@@ -113,7 +113,7 @@ The active inference service is `frank-archon.service` (`systemctl is-active`
 `active`, `is-enabled` → `enabled`, read 2026-10-18): radiance 1.0.13 in a
 host-network podman container answering `curl 127.0.0.1:7770/health` →
 `{"status":"ok","model":"frank-archon",…}`. See
-[[frank-archon-radiance-qwen38-iq4r]].
+[[frank-archon]].
 
 The services this page once called active are gone: `frank-medium.service`
 (llama-server, 32:33 layer split + DFlash2 drafter) is `not-found` — that rung
@@ -146,7 +146,7 @@ unverified. See [[deployment]].
 
 ## See Also
 
-- [[frank-archon-radiance-qwen38-iq4r]] — the active serving runbook
+- [[frank-archon]] — the active serving runbook
 - [[Dual-R9700-Cooperation]] — why the PCIe path matters, and what the swap changed
 - [[deployment]] — the deploy flow
 - [[Shared Memory Architecture Dankclaw]] — for contrast, Dankclaw's unified memory

serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md

@@ -16,7 +16,7 @@ timestamp: 2026-09-16
 
 Dankserver's two R9700s run **one tensor-parallel engine across both cards** —
 today that is `frank-archon` under radiance 1.0.13, TP=2
-([[frank-archon-radiance-qwen38-iq4r]]). When this page was written the endpoint
+([[frank-archon]]). When this page was written the endpoint
 was `frank-medium`: first llama.cpp's layer split
 ([[frank-medium-llamacpp-dflash2]], 2026-09-16 → 09-20), then from 2026-09-20
 one vLLM-radiance process at `tensor_parallel_size 2`
@@ -51,7 +51,7 @@ What that bought:
   decode and 3.6x at 4-way concurrency, and it replaced the layer split on the day
   of measurement. The payload has moved since: the MXFP4 `frank-medium` shape is
   retired 2026-09-26, and what runs today is the iq4r publisher under radiance
-  1.0.13 ([[frank-archon-radiance-qwen38-iq4r]]).
+  1.0.13 ([[frank-archon]]).
 - **The drafter, not the collective, is what llama.cpp lost.** Its tensor mode
   could not carry the DFlash2 drafter, which is the honest caveat on the
   comparison below.

serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md

@@ -24,7 +24,7 @@ timestamp: 2026-09-03
 > usable again since the 2026-10-18 `iommu=pt` boot (`amdxdna` bound,
 > `/dev/accel/accel0` present — [[dankclaw-iommu-translation-off]]), but no
 > service runs on it. `frank-zealot` today is the llama.cpp PrismML endpoint on
-> the RTX 3090 ([[frank-zealot-llama-prism-bonsai2-pq20]]). See [[frank-npu]]
+> the RTX 3090 ([[frank-zealot]]). See [[frank-npu]]
 > for the retirement record. Everything below is the 2026-08 research record.
 
 # FastFlowLM NPU Inference

serverAppConfigs-docs/LLM/GGUF Compatibility.md

@@ -119,5 +119,5 @@ re-download.
 
 - [[Llama Cpp Runtimes]] — which fork ships which custom types
 - [[Llama Cpp Code Diff]] — the enum tables and code-level differences
-- [[frank-dragoon-rocmfpx-q8agent-mtp]] — the ftype-115 lock-in in production
+- [[frank-dragoon]] — the ftype-115 lock-in in production
 - [[Llama Cpp Serving Sizing]] — pricing what a payload costs once it loads

serverAppConfigs-docs/LLM/Hybrid CPU GPU Processing Dankserver.md

@@ -10,7 +10,7 @@ timestamp: 2026-08-28
 
 ## Overview
 
-Dankserver has **no NVIDIA GPU**: the RTX 3090 left the box on 2026-09-15 and the host now runs two AMD Radeon AI PRO R9700s (`lspci | grep -ci nvidia` → 0, read 2026-10-18 — [[Dankserver System Specs]]). This page's vLLM-era hybrid guidance is reference material, not the serving path: the live service is radiance 1.0.13 as `frank-archon`, and its hybrid mechanism is a pinned host pool (43.88 GiB per rank, `scripts/frank-archon.env:184`) plus the CPU-resident 47.68 GiB n-gram/PLE table, not vLLM KV swapping ([[frank-archon-radiance-qwen38-iq4r]]).
+Dankserver has **no NVIDIA GPU**: the RTX 3090 left the box on 2026-09-15 and the host now runs two AMD Radeon AI PRO R9700s (`lspci | grep -ci nvidia` → 0, read 2026-10-18 — [[Dankserver System Specs]]). This page's vLLM-era hybrid guidance is reference material, not the serving path: the live service is radiance 1.0.13 as `frank-archon`, and its hybrid mechanism is a pinned host pool (43.88 GiB per rank, `scripts/frank-archon.env:184`) plus the CPU-resident 47.68 GiB n-gram/PLE table, not vLLM KV swapping ([[frank-archon]]).
 
 ## Architecture
 

serverAppConfigs-docs/LLM/Llama Cpp Serving Sizing.md

@@ -87,7 +87,7 @@ co-tenants.
 (a retired frank-zealot shape: 22.6 of 24.1 GiB for an 11.29 GiB model; the card
 today reads 24,576 MiB = 24.0 GiB total, `nvidia-smi --query-gpu=memory.total`,
 2026-10-18, and the deployed payload is the PQ2_0 pair
-[[frank-zealot-llama-prism-bonsai2-pq20]]). Do the capacity
+[[frank-zealot]]). Do the capacity
   subtraction before designing: 12 attn layers × 1.62 GiB + 200k f16 KV =
   28.6 GiB > 24.1 GiB usable → KV-in-VRAM on that pair only reaches ~64k.
 - **Context lengths: derive, never remember.** `llama-bench` computes
@@ -168,7 +168,7 @@ in the timings. Context cap for full-model residency on that shape stayed
 ## Related
 
 - [[GGUF Compatibility]] — what a payload is before it is priced
-- [[frank-zealot-llama-mimo-q6k-2x200704]] — the two-boot priced shape of the retired MiMo era (live shape today: [[frank-zealot-llama-prism-bonsai2-pq20]])
+- [[frank-zealot]] — the two-boot priced shape of the retired MiMo era (live shape today: [[frank-zealot]])
 - [[frank-medium-llamacpp-dflash2]] — where the 68 KiB/token datum was measured
 - [[DFlash DSpark Speculative Decoding]] — the spec A/B protocol on llama.cpp
 - [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — the multi-device side

serverAppConfigs-docs/LLM/Model Swap and Rollback.md

@@ -285,4 +285,4 @@ predecessor "stopped" (the port race above).
 - [[Llama Cpp Serving Sizing]] — pricing the shape once it loads
 - [[RDNA4 MoE Expert Offload VRAM Budget]] — vLLM-side fit arithmetic
 - [[LLM Bench Windows and Measurement]] — measuring the new deployment
-- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the retired tcclaviger vLLM endpoint (2026-09-29 → 10-05); the live Dankserver endpoint is [[frank-archon-radiance-qwen38-iq4r]]
+- [[frank-archon]] — the retired tcclaviger vLLM endpoint (2026-09-29 → 10-05); the live Dankserver endpoint is [[frank-archon]]

serverAppConfigs-docs/LLM/PCIe Link and Atomics.md

@@ -253,4 +253,4 @@ kernel's declared args (`roc-obj-extract` + `llvm-readelf --notes`), look for
 
 - [[Llama Cpp Multi-Device Splits]] — the layer-split path around the wall
 - [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — measured history
-- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the gather economics of the retired tcclaviger vLLM era (2026-09-29 → 10-05)
+- [[frank-archon]] — the gather economics of the retired tcclaviger vLLM era (2026-09-29 → 10-05)

serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.8 PP Deployment.md

@@ -17,7 +17,7 @@ timestamp: 2026-09-03
 > removed from Dankserver on 2026-09-15 (it now sits in Dankclaw) and the server
 > runs 2× R9700. The live multi-GPU scheme on Dankserver is tensor parallelism
 > inside one process ([[Dual-R9700-Cooperation]],
-> [[frank-archon-radiance-qwen38-iq4r]]).
+> [[frank-archon]]).
 
 # Qwen3.8-27B Pipeline-Parallel Deployment (Dankclaw + Dankserver)
 
@@ -199,7 +199,7 @@ exists** (checkout read 2026-10-18: `frank-pp-worker.env`,
 `serve-pp-worker.sh`, `serve-pp-head.sh`, `frank-pp-worker.service`,
 `pp-test.sh`, `pp-bench.py` and `prune-rpc-cache.sh` are all absent), and
 `scripts/frank-dragoon.env` is today the halogen-flash-server env, not a PP
-head env ([[frank-dragoon-flash-server-v2-native]]).
+head env ([[frank-dragoon]]).
 
 ## Gotchas (all verified in b10433 source / live tests, 2026-08-15)
 

serverAppConfigs-docs/LLM/Pipeline Parallelism/index.md

@@ -65,6 +65,6 @@ graph LR
 ## See Also
 
 - [[DFlash DSpark Speculative Decoding]]
-- [[frank-zealot-benchmark]]
+- [[frank-zealot]]
 - [[Dankserver System Specs]]
 - [[Dankclaw System Specs]]

serverAppConfigs-docs/LLM/Quantization Production.md

@@ -253,8 +253,8 @@ top-8**, moe_intermediate 512, hidden 2048, MTP ×1, 262k ctx, vocab
 - Repo: `scripts/convert-swift15-w4dn8.sh`,
   `scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`,
   `scripts/kld-gate-swift15.sh`, `scripts/build-llama-rdna4.sh`,
-  `scripts/build-kairic-edge-gfx1151.sh`, [[agentionai-fast-imatrix]],
+  `scripts/build-kairic-edge-gfx1151.sh`, [[frank-dragoon]],
-  [[kingjones-strix-rocmfp4]], [[qwen3.8-strix-optimizations]],
+  [[frank-dragoon]], [[qwen3.8-strix-optimizations]],
-  [[frank-archon-radiance-swift15-w4dn8]],
+  [[frank-archon]],
-  [[frank-zealot-llama-qwen35-9b-q6k-2x200704]],
+  [[frank-zealot]],
-  [[frank-dragoon-flash-server-v2-swift15]], [[GGUF Compatibility]].
+  [[frank-dragoon]], [[GGUF Compatibility]].

serverAppConfigs-docs/LLM/RDNA4 Container Serving.md

@@ -13,7 +13,7 @@ Bring-up path for Dankserver (2x Radeon AI PRO R9700, gfx1201, 31.86 GiB each =
 `docker.io/tcclaviger/vllm` (vLLM 0.29.0.dev, GLM/Qwen-specialised RDNA4 build
 with native MoE quant kernels and a first-class expert offload planner). The
 tuning history and current deployed shape live in
-[[dankserver-qwen38-fp8-expert-offload-tuning]]; this page is the bring-up law.
+[[frank-archon]]; this page is the bring-up law.
 
 ## 0. Know the box — and its phantom third GPU
 
@@ -172,7 +172,7 @@ the REMAINDER to the expert arena. Consequences, all measured:
 - **`--expert-offload-mem` caps the HOST side, not the VRAM arena**
   (`cap = int(x * GIB)` against `host_memory_available()`). Do not reach for
   it to fix a VRAM problem; to grow the arena, shrink the KV pin or raise
-  util (see [[dankserver-qwen38-fp8-expert-offload-tuning]] for the
+  util (see [[frank-archon]] for the
   util=1.0 unbootability law).
 
 **Read the plan — it is the real acceptance test:**
@@ -238,13 +238,13 @@ sequential STEPS rather than making each cheaper.
 boolean (default 0; at depth 1 only ~10 % cold-vs-warm difference — the cache
 already works); `CLAV_TUNABLEOP_SWEEP=1` autotunes GEMM shapes (silently
 skipped under expert offload unless `CLAV_TUNABLEOP_WITH_OFFLOAD=1` too —
-see [[dankserver-qwen38-fp8-expert-offload-tuning]] for why it was rejected);
+see [[frank-archon]] for why it was rejected);
 `--expert-precise-memory` swaps the fitted overhead figure in.
 
 ## Related
 
-- [[dankserver-qwen38-fp8-expert-offload-tuning]] — deployed shape, plan law,
+- [[frank-archon]] — deployed shape, plan law,
   measured campaigns
 - [[PCIe Link and Atomics]] — why decode is gather-bound and what cannot fix it
 - [[RDNA4 MoE Expert Offload VRAM Budget]] — arena/plan arithmetic deep-dive
-- [[frank-dragoon-benchmark]] and [[frank-zealot-benchmark]] — window discipline on live hosts
+- [[frank-dragoon]] and [[frank-zealot]] — window discipline on live hosts

serverAppConfigs-docs/LLM/RDNA4 MoE Expert Offload VRAM Budget.md

@@ -11,7 +11,7 @@ timestamp: 2026-10-05T00:00:00-07:00
 For VRAM-bound vLLM on the tcclaviger RDNA4 fork (frank-archon: Qwen3.8-
 Flash-Next FP8, TP=2, 2x R9700 at 31.86 GiB each, expert offload to host RAM).
 Tuning history and the deployed shape live in
-[[dankserver-qwen38-fp8-expert-offload-tuning]]; boot-failure capture lives in
+[[frank-archon]]; boot-failure capture lives in
 [[vLLM Boot Forensics and Config Interrogation]].
 
 ## 1. The plan law
@@ -215,5 +215,5 @@ at graph capture; 6 seqs fp8 → 10.70 GiB ask with 9.18 GiB free (fails) —
 
 - [[RDNA4 Container Serving]] — the bring-up path and container traps
 - [[vLLM Boot Forensics and Config Interrogation]] — capture and contracts
-- [[dankserver-qwen38-fp8-expert-offload-tuning]] — measured history
+- [[frank-archon]] — measured history
 - [[Llama Cpp Serving Sizing]] — the llama.cpp equivalent

serverAppConfigs-docs/LLM/Shared Memory Architecture Dankclaw.md

@@ -233,7 +233,7 @@ FP8 quantization depends on GPU architecture:
 
 For FP4 on Dankclaw the ROCmFPX llama.cpp fork (`Q2_0_ROCMFP4`) provided native
 ROCm FP4 kernels; that runtime line is **retired** (CIRU/Kairic eras,
-[[ciru-activefpx-promptforge]], [[frank-dragoon-kairic-edge-qwen38-27b]]) and no
+[[frank-dragoon]], [[frank-dragoon]]) and no
 ROCmFPX build serves this host today.
 
 ## Example Commands

serverAppConfigs-docs/LLM/VLLM Index.md

@@ -9,7 +9,7 @@ timestamp: 2026-08-28
 
 # vLLM Index
 
-> **RETIRED (2026-08-18).** The vLLM service on Dankclaw was replaced by the CIRU ROCmFPX llama.cpp runtime (v3). These pages are historical reference. See [[ciru-activefpx-promptforge]].
+> **RETIRED (2026-08-18).** The vLLM service on Dankclaw was replaced by the CIRU ROCmFPX llama.cpp runtime (v3). These pages are historical reference. See [[frank-dragoon]].
 
 ## Documentation Overview
 

serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md

@@ -1,217 +0,0 @@
----
-type: reference
-title: agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw
-description: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF served by the charlie12345/ROCmFPX llama.cpp runtime (build-qwen38). 87.06 GiB Q4_0_ROCMFP4_FAST imatrix (v2 joined PLE layout), own MTP draft head (ROCmFP4-FAST, 2.28 GiB). frank-dragoon.
-tags:
-  - llama.cpp
-  - rocm
-  - dankclaw
-  - gfx1151
-  - agentionai
-  - rocmfp4
-  - strix-halo
-  - qwen4exp
-  - mtp
-timestamp: 2026-08-29
----
-
-# agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw
-
-> **RETIRED 2026-08-30** — replaced by the kingjones STRIX model on the
-> card's pinned kingjones30/ROCmFPX runtime; see
-> [[kingjones-strix-rocmfp4]] for the active runbook. This page stays as
-> the deployment record: the v2-joined-PLE layout requirement, the MTP
-> head incompatibility (verified 3x), and the QSA F32-mask patch
-> (`scripts/patches/qwen4exp-qsa-f32-mask.patch`) that made
-> `--flash-attn on` work on the charlie build-qwen38 tree — all still
-> true for that binary + model pair, which remains the rollback target.
-
-`frank-dragoon` (port 7770) served
-`agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF` (main,
-`Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf`, the `v2/` joined layout)
-with the author's own MTP draft head
-(`Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf`, never loaded) on the
-charlie12345/ROCmFPX `build-qwen38` runtime, 2026-08-29 → 2026-08-30.
-
-## Why this model
-
-- 87.06 GiB at 4.23 bpw — everything on the GPU of the 128 GB unified-memory
-  box, including the 51.2 B-parameter n-gram (PLE) table. REPO `v2/` layout
-  is used (joined single `per_layer_token_embd.weight` tensor) because the
-  charlie12345/ROCmFPX loader requires it; the root `-ple16` file's per-head
-  split only loads in the LaurentZuijdwijk Vulkan fork (see Known issues #2).
-- imatrix-calibrated on 1540 chunks; +2.48% PPL vs reference on wikitext-2
-  (better than AesSedai IQ4_XS / IQ3_S at smaller size).
-- 18-25% faster prompt processing than a mainline quant of similar size
-  (356 vs 285 t/s at 2048 tok; 301 vs 256 at 8k depth).
-- Ships the model's own f16 vision tower (`mmproj/`).
-- The MTP head is quantized to match the target (ROCmFP4-FAST) — a Q8_0
-  draft measured worse on this head family — but it does not load on
-  build-qwen38 (Known issues #3).
-
-## Runtime
-
-charlie12345/ROCmFPX `llama-server` built at
-`/home/dank/GitPackages/ROCmFPX/build-qwen38` (HIP, ROCm 7.2.4, gfx1151,
-PR #27742 qwen4exp merged). Upstream llama.cpp cannot load the ROCmFP4
-quant types (`TYPE_100`/`TYPE_101`/FAST variants) — verified:
-
-```
-strings build-qwen38/bin/llama-server | grep Q4_0_ROCMFP4_FAST
-```
-
-CIRU binaries (ciru-ai v1.1) do **not** carry the FAST types; do not point
-this model at the CIRU runtime. The build previously served the kingjones
-STRIX / STRIX_LEAN models; the agentionai-FAST pair is the first use of the
-author's own matched MTP head on this binary.
-
-## Files / download
-
-| Item | Path on Dankclaw | Size |
-|---|---|---|
-| Main (v2 joined) | `/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/v2/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf` | 93,484,242,816 B (87.06 GiB) |
-| Main (root ple16, unused) | `/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf` | 93,484,237,760 B (87.06 GiB) |
-| Vision | `…/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf` | 904,004,128 B |
-| Draft head | `/home/dank/models/mtp-Qwen3.8-Flash-Next-FAST/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf` | 2,444,519,296 B |
-
-Draft head sha256 (verified against HF LFS):
-`5046d69571bb35c699e19c7a00b36633c2fdfdd06d70482551744ec523a6b590`
-
-Downloaded with a resumable plain `curl -L -C -` (Hugging Face `resolve/main`
-URLs; no auth token needed for public repos). The mmproj lives in an `mmproj/`
-subdirectory of the repo.
-
-## Serving profile
-
-Card sampling table (instruct, non-thinking) mixed with the validated
-build-qwen38 profile:
-
-```
-llama-server \
-  --model Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf \
-  --mmproj mmproj-Qwen3.8-Flash-Next-f16.gguf \
-  --host 0.0.0.0 --port 7770 --jinja \
-  --device ROCm0 --gpu-layers 999 \
-  --split-mode none --fit off --flash-attn on \
-  --ctx-size 32768 --parallel 1 \
-  --batch-size 4096 --ubatch-size 4096 \
-  --cache-type-k f16 --cache-type-v f16 \
-  --threads 16 --threads-batch 16 \
-  --ctx-checkpoints 16 --cache-ram 8192 \
-  --cache-prompt --cache-idle-slots --metrics \
-  --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 \
-  --reasoning on --reasoning-format deepseek --reasoning-budget -1 \
-  --chat-template-file chat-template.jinja \
-  --chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}' \
-  --spec-type ngram-mod
-```
-
-Key choices and deviations from the CIRU deployment:
-
-- **`--fit off`** — auto-fit segfaults on qwen4exp hyper-connection config
-  (same as every Flash-Next deployment).
-- **`--checkpoint-min-step` removed** — build-qwen38 does not implement it
-  (verified via `--help`); only `--ctx-checkpoints` is passed.
-- **No PLE sidecar args** — the n-gram table is in-GGUF (joined
-  `per_layer_token_embd.weight` from the `v2/` layout), not a `ple/`
-  CIRUPLE1 sidecar. `--ple-sidecar`/`--ple-cache-mib` and the
-  `GGML_QWEN4EXP_PLE_*` / `GGML_CUDA_Q41_MOE_FORCE_J` env knobs are CIRU-
-  runtime-only and are gone.
-- **`--mmproj` active** — vision tower included (f16, lossless).
-- **`--flash-attn on` (restored 2026-08-29).** fa on previously aborted graph
-  build: the QSA indexer path (layers with attention compression) called
-  `ggml_fill` on the F16-cast kq mask (`GGML_ASSERT(a->type == GGML_TYPE_F32)`,
-  ggml.c:5371, backtrace through `build_attn_qsa`). Fixed in the fork source
-  by the QSA F32-mask patch — vendored at
-  `scripts/patches/qwen4exp-qsa-f32-mask.patch` (sha256
-  `f2f4a486e21cfad80bfa42683281d18e1cba22fa43c5e0e2aa0cfdbe141e925f`);
-  the binary at `build-qwen38/bin/llama-server` (rebuilt 2026-08-29) already
-  carries it. A clean rebuild must re-apply
-  `git apply scripts/patches/qwen4exp-qsa-f32-mask.patch` inside
-  `/home/dank/GitPackages/ROCmFPX` before `make`. Verified live at
-  CTX_SIZE=32768: model loads, health 200, real completion.
-- Context capped at **32768 × 1 slot — the author's own quick-start**. This
-  model is fully GPU-resident (87.06 GiB): the PLE n-gram table sits inside
-  the GGUF as one tensor, unlike STRIX_LEAN which streamed it from RAM.
-  HIP heap is 119.3 GiB; the context buffer is 50.8 GiB at 131072 and
-  103 GiB at 262144, so either window OOMs (`cudaMalloc … failed: out of
-  memory`, verified live on first graph alloc). At 32768 the buffer is
-  ~13 GiB (~100 GiB total) and loads clean.
-- MTP is disabled (ngram-mod only) — see the MTP section below.
-
-## MTP draft head — INCOMPATIBLE on build-qwen38
-
-The author's own matched head (`agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-
-FAST-GGUF`, 2.28 GiB, `qwen4exp` arch) does **not** load on this binary:
-
-```
-error loading model: missing tensor 'blk.0.hc_attn_norm.weight'
-```
-
-The build-qwen38 qwen4exp loader requires a full per-layer
-`blk.<il>.hc_attn_norm.weight` set in the draft (the graph reads all 48
-layers); the head only carries `blk.48.hc_attn_norm.weight` (count 1 — all
-three Flash-Next-era heads examined on this fork have that same partial
-layout). Verified three times, not a cross-vendor artifact:
-
-1. EasiiX Q8_0 head vs kingjones STRIX_LEAN main (2026-08-28) — failed.
-2. agentionai FAST head vs kingjones STRIX_LEAN main (2026-08-28) — failed.
-3. agentionai FAST head vs agentionai v2 main (2026-08-29, this deployment)
-   — failed identically.
-
-The 2.28 GiB head stays on disk; the moment a binary or head appears that
-carries the full per-layer norm set, re-enable with
-`SPEC_TYPE=draft-mtp,ngram-mod`, `--spec-draft-n-max 3` (author-measured
-best: 32.4 t/s, 0.612 acceptance; adaptive p-split measured worse).
-
-## Config files
-
-- Env: `scripts/frank-dragoon.env`
-- Serve: `scripts/serve-frank-dragoon.sh`
-- Chat template: `scripts/chat-template.jinja` (unchanged, Qwen-Sharp)
-- Fork patch: `scripts/patches/qwen4exp-qsa-f32-mask.patch` (re-apply after
-  any clean fork rebuild — see Serving profile)
-
-## Swap procedure
-
-```bash
-sudo systemctl stop frank-dragoon
-# edit scripts/frank-dragoon.env + scripts/serve-frank-dragoon.sh, commit, push
-# on Dankclaw: git pull --ff-only && scripts/deploy.sh dankclaw --service frank-dragoon
-sudo systemctl start frank-dragoon   # or via deploy.sh
-curl http://192.168.0.143:7770/health
-```
-
-Model load is slow (87 GiB + 2.3 GiB draft): health returns 503 while
-loading; `deploy.sh` waits up to 20 minutes (`LLM_WAIT`). Do not kill it.
-
-## Known issues
-
-1. **ROCmFP4 types require the ROCmFPX fork.** Stock llama.cpp and the CIRU
-   runtime reject these files.
-2. **Use the `v2/` joined layout, NOT the root `-ple16` split.** This binary's
-   qwen4exp loader (qwen4exp.cpp:104-108) unconditionally requires a single
-   `per_layer_token_embd.weight` tensor and GGML_ASSERTs "qwen4exp is missing
-   the PLE n-gram table" against the per-head split. The root file only loads
-   in the LaurentZuijdwijk Vulkan fork (`vulkan-qwen4exp-rocmfpx`). The v2
-   file is byte-identical in weights, ~5 KB larger header, loads fully into
-   SVM unified memory (no `--ngram-on-disk` needed — kingjones STRIX ran a
-   50 GiB joined PLE table the same way).
-3. **No MTP draft head loads on build-qwen38.** The qwen4exp loader
-   requires a full per-layer `blk.<il>.hc_attn_norm.weight` set; every
-   Flash-Next-era head checked (EasiiX Q8_0, agentionai FAST, CIRU) carries
-   only `blk.48`. Spec decoding is ngram-mod only. See the MTP section.
-4. **`--flash-attn on` (restored 2026-08-29).** fa on aborts graph build
-   without the QSA F32-mask patch — see Serving profile (fix + sha256
-   vendored there).
-5. **`SVM mapping failed` kernel spam** is a documented symptom, not root
-   cause — investigate before blaming memory pressure.
-
-## Citations
-
-- [agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF)
-- [agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF)
-- [Qwen/Qwen3.8-Flash-Next base model](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)
-- [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) — fork with PR #27742 (qwen4exp architecture) and the ROCmFP4 types
-- [llama.cpp PR #27742](https://github.com/ggml-org/llama.cpp/pull/27742) — qwen4exp support (merged upstream)
-- [llama.cpp PR #27739](https://github.com/ggml-org/llama.cpp/pull/27739) — MTP draft head support

serverAppConfigs-docs/LLM/ciru-activefpx-promptforge.md

@@ -1,266 +0,0 @@
----
-type: reference
-title: CIRU ActiveFPX PromptForge on Dankclaw
-description: jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (v3) served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Install runbook, ROCm 7.15 toolchain, and the frank-dragoon service.
-tags:
-  - llama.cpp
-  - rocm
-  - dankclaw
-  - gfx1151
-  - ciru
-  - promptforge
-timestamp: 2026-08-22
----
-
-# CIRU ActiveFPX PromptForge on Dankclaw
-
-> **RETIRED 2026-08-24.** `frank-dragoon` served this v3 model from
-> 2026-08-22 (restored after a same-day Kairic Edge IU4 experiment was
-> reverted — operator decision, commit `c60156f`; see [[kairic-edge]])
-> until [[eemin-rocmi4-mtp]] took over on 2026-08-24. Kept as the
-> historical record of the v3 deployment.
-
-`frank-dragoon` (port 7770, moved from 52625 on 2026-08-22) serves
-`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` **v3** through the CIRU
-ROCmFPX llama.cpp runtime. This replaced the retired vLLM service that
-previously occupied the slot. The v2 deployment moved to v3 on 2026-08-21.
-
-## What This Is
-
-The model is a **GGUF**, not a vLLM checkpoint. v3 is the consolidated
-quality-and-speed release of Qwen3.8-27B for AMD ROCm: an FFN-equalized
-compact body, 50 retained late-writer Q6 tensors, full-Q8 output
-projection, full-W8 PromptForge view (FFN + Gated DeltaNet), the
-small-row W8 route, a draft-only Output-K8 shortlist proxy with Q8
-reranking, native depth-4 MTP (no ngram), and the BF16 Qwen3.8 vision
-tower + 5120-wide merger as a separate projector.
-
-| Artifact | Size |
-|----------|------|
-| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v3-Q8-FFNEQ-LateQ6.gguf` | 15.44 GiB |
-| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-FFN.pfs` | 15.95 GiB |
-| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-GDN.pfs` | 3.75 GiB |
-| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-Output-K8.pfs` | 0.67 GiB |
-| `Qwen3.8-27B-mmproj-BF16.mmproj` | 0.87 GiB |
-| Full v3 payload (GGUF + 3 .pfs + mmproj) | ~36.7 GiB |
-
-The `.pfs` companions and the Output-K8 proxy are optional. Omit them to
-run the GGUF alone and save ~20.6 GiB of memory. The GGUF remains
-independently usable.
-
-## Requirements
-
-- **ROCm 7.15 is required** for the advertised Strix Halo performance.
-  The system ROCm (`/opt/rocm-7.2.4`) selects slower or fallback kernels.
-- The pinned runtime is `ciru-ai/ROCmFPX` tag
-  `qwen3.8-activefpx-promptforge-v2.3` (commit `103af8b41`) **plus the
-  v3 runtime patch** `runtime/qwen38-v3-output-k8-runtime.patch`. On
-  Dankclaw the patch is commit `af0eac6` on top of the v2.3 checkout.
-- Composable Kernel at `fdf4bb7fcc984811cef48ce817d89aac064b984a`.
-- Validated GPU: AMD Radeon 8060S / gfx1151.
-
-## ROCm 7.15 Toolchain
-
-The retired `ling3-ciru` package left a complete TheRock ROCm 7.15 SDK
-(`7.15.0a20260728`) inside its venv:
-
-```
-/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/
-  _rocm_sdk_devel/      # hipcc, amdclang 23, cmake, headers
-  _rocm_sdk_libraries/  # hipblas, hipblaslt, rocroller
-  _rocm_sdk_core/       # libamdhip64, libhsa-runtime64
-  rocm-sdk-device-gfx1151
-```
-
-The compiler is amdclang 23. `rocm-sdk version` reports `7.15.0a20260728`.
-The binary is built against and run with these 7.15 libraries.
-
-## Build Runbook (2026-08-18, v2.3; +v3 patch on 2026-08-21)
-
-Step 1-4 below built v2.3 on 2026-08-18. On 2026-08-21 the v3
-Output-K8 runtime patch was applied on top and the server rebuilt:
-
-0. Apply the v3 runtime patch to the v2.3 checkout and rebuild:
-   ```
-   % verify the patch hash: 8ba7d721ea380b060aeb0c7595b8b05f1582c0f7bf7a4b89e0910bc8e72465d5
-   git am runtime/qwen38-v3-output-k8-runtime.patch   # -> commit af0eac6
-   cmake --build build-promptforge --target llama-server -j$(nproc)
-   ```
-   The pre-built binary is kept as `build-promptforge/bin/llama-server.v2.3-bak`
-   for rollback.
-
-1. Install host build tools (ninja-build was missing): `sudo apt install ninja-build`.
-2. Clone the runtime and CK at their pinned revisions:
-   ```
-   git clone --depth 1 --branch qwen3.8-activefpx-promptforge-v2.3 \
-     https://github.com/ciru-ai/ROCmFPX.git
-   git clone --depth 1 https://github.com/ROCm/composable_kernel.git
-   git -C composable_kernel checkout fdf4bb7fcc984811cef48ce817d89aac064b984a
-   ```
-3. CK headers are generated at CK configure time (`ck/config.h`,
-   `ck/version.h`). The llama.cpp build includes the CK source tree, so run
-   CK's configure once and copy the headers into `include/ck/`:
-   ```
-   cmake -S composable_kernel -B build-ck -DAMDGPU_TARGETS=gfx1151 -DCK_ENABLE_WMMA=ON
-   cp build-ck/include/ck/config.h build-ck/include/ck/version.h composable_kernel/include/ck/
-   ```
-4. Configure the llama.cpp build against the venv ROCm 7.15 SDK:
-   ```
-   export ROCM_PATH=$SDK/_rocm_sdk_devel
-   cmake -S ROCmFPX -B build-promptforge -G Ninja \
-     -DCMAKE_BUILD_TYPE=Release \
-     -DCMAKE_C_COMPILER=$ROCM_PATH/llvm/bin/clang \
-     -DCMAKE_CXX_COMPILER=$ROCM_PATH/llvm/bin/clang++ \
-     -DCMAKE_HIP_COMPILER=$ROCM_PATH/llvm/bin/clang++ \
-     -DCMAKE_PREFIX_PATH=$ROCM_PATH \
-     -DGGML_HIP=ON -DGGML_CUDA=OFF -DGGML_VULKAN=OFF \
-     -DGGML_HIP_FORCE_MMQ=ON -DGGML_HIP_GRAPHS=ON \
-     -DGGML_HIP_MMQ_MFMA=ON -DGGML_HIP_NO_VMM=ON \
-     -DGGML_HIP_ROCWMMA_FATTN=OFF -DGGML_NATIVE=ON \
-     -DAMDGPU_TARGETS=gfx1151 -DGPU_BUILD_TARGETS=gfx1151 \
-     -DPROMPTFORGE_CK_ROOT=$CK_ROOT \
-     -DLLAMA_BUILD_SERVER=ON \
-     -DLLAMA_BUILD_WEBUI=OFF
-   cmake --build build-promptforge --target llama-server -j$(nproc)
-   ```
-   `LLAMA_BUILD_WEBUI=OFF` is required. The embedded browser UI downloads
-   assets from an HF bucket (`llama-ui`) that returns 404; the service is
-   consumed through the OpenAI-compatible API, so the UI is not needed.
-
-Build output: `build-promptforge/bin/llama-server`.
-
-```mermaid
-graph TD
-    A[Clone ROCmFPX v2.3 + CK pinned] --> B[Configure CK, copy generated headers]
-    B --> C[Configure llama-server vs ROCm 7.15 SDK]
-    C --> D[Build llama-server]
-    D --> E[Verify --list-devices shows gfx1151]
-    E --> F[Download v3 GGUF + FFN/GDN/Output-K8 + mmproj]
-    F --> G[Deploy frank-dragoon]
-    G --> H[Health check + chat completion]
-```
-
-## Model Download
-
-With the HF token to the v3 staging directory
-`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/`. The v3 files live there
-so the v2 artifacts in `/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX/` stay
-intact for rollback. Download ran on Dankclaw with the background script
-`/tmp/download-v3.sh`; all five files + the patch passed SHA-256 checks
-against the model card before the service restart.
-
-## Service
-
-- Unit: `frank-dragoon.service`, port 7770
-- Script: `scripts/serve-frank-dragoon.sh`, env `scripts/frank-dragoon.env`
-- The script exports the ROCm 7.15 SDK libraries in `LD_LIBRARY_PATH` before
-  the system ROCm. `ROCM_PATH` points at the devel SDK.
-
-### Launch profile
-
-- 262144 context (native max), 1 slot (user request 2026-08-19; parallel
-  slots degraded performance), f16 KV (~16 GiB), context checkpoints (16),
-  prompt cache. Concurrent requests queue on the single slot; the server
-  never preempts a running generation
-- Native MTP draft, depth 4, device ROCm0
-- `--temp 0.7 --top-p 0.95`, reasoning on (`--reasoning-format deepseek`
-  puts `<think>` reasoning in `message.reasoning_content`)
-- v3 runtime: Output-K8 draft proxy (top-8 shortlist, Q8 rerank) +
-  small-row W8 prompt route, full-W8 FFN/GDN views
-- PromptForge env: `PROMPTFORGE_MODE=m2048_fused_tail1476`,
-  `GGML_CUDA_GRAPH_OPT=0`, `PROMPTFORGE_MTP_OUTPUT_K8=1`,
-  `PROMPTFORGE_MTP_OUTPUT_K8_VALIDATE=0`,
-  `PROMPTFORGE_ENABLE_SMALLM_W8=1`
-
-### Required environment
-
-```
-export PROMPTFORGE_SIDECAR=...FFN.pfs
-export PROMPTFORGE_GDN_SIDECAR=...GDN.pfs
-export PROMPTFORGE_MODE=m2048_fused_tail1476
-export GGML_CUDA_GRAPH_OPT=0
-export PROMPTFORGE_ENABLE_SMALLM_W8=1
-export PROMPTFORGE_MTP_OUTPUT_K8=1
-export PROMPTFORGE_MTP_OUTPUT_K8_VALIDATE=0
-export PROMPTFORGE_MTP_OUTPUT_K8_PROXY=...Output-K8.pfs
-# v3 qualified profile: disable the CPU argmax draft fast path and any
-# IU4/ngram lane. The Output-K8 proxy + backend sampling replace them.
-unset LLAMA_MTP_CPU_ARGMAX_FASTPATH LLAMA_MTP_BACKEND_GREEDY
-unset PROMPTFORGE_IU4_SIDECAR PROMPTFORGE_GDN_IU4_SIDECAR
-unset PROMPTFORGE_ENABLE_NGRAM_MOD PROMPTFORGE_ENABLE_NGRAM_M65_IU4
-```
-
-The v3 qualified profile runs the MTP draft through the Output-K8 proxy,
-not the CPU argmax fast path. The old `LLAMA_MTP_CPU_ARGMAX_FASTPATH=1`
-profile is retired for this slot. A TOP_K sampler warning may still appear
-in the journal once per start; the Output-K8 draft path does not depend on
-it and generation proceeds.
-
-## Operational Notes
-
-- Health: `curl http://127.0.0.1:7770/health`. Returns 503 while loading.
-- Model load is slow (v3 five-file payload ~36.7 GiB); do not kill the
-  process during warmup. The page cache makes a repeat load fast right after
-  download, but a later cold start takes several minutes.
-- `deploy.sh dankclaw` waits 20 minutes for health.
-- The systemd unit must not set `HOST` or `TEMP` env vars (HIP collision
-  rule). `--host`/`--port`/`--temp` are CLI flags only.
-- v3 adds the BF16 vision tower over v2. The service reports `multimodal`
-  capability; mmproj loads with the language model (CLIP on ROCm0 backend).
-
-## ngram-mod + MTP Stacking (benchmarked 2026-08-21)
-
-This page records a research question: does ngram-mod add value when
-stacked under the native MTP draft? The v3 profile runs MTP alone; the
-benchmark below compares that against the stacked pair.
-
-### Feasibility
-
-ngram-mod DOES stack with MTP in this CIRU fork. The v3 build compiles in
-both `ngram-mod` and `draft-mtp` (`common/speculative.cpp`). Pass
-`--spec-type draft-mtp,ngram-mod` (or `ngram-mod,draft-mtp`) and the fork
-initializes both implementations in the same process. The boot log shows:
-
-```
-adding speculative implementation 'ngram-mod'
-initialized ngram_mod with n_match=24, size=4194304 (16.000 MB)
-adding speculative implementation 'draft-mtp'
-```
-
-The model card's "no ngram subsystem" line refers to the CIRU PromptForge
-env lanes (`PROMPTFORGE_*_NGRAM_*`, `PROMPTFORGE_ENABLE_NGRAM_MOD`), not
-the upstream llama.cpp spec chain. Those env lanes must stay unset; the
-`--spec-type ngram-mod` chain is separate and works.
-
-### Ordering — the CLI order does not matter
-
-The `--spec-type` string order is discarded. `common_speculative_init`
-collapses the requested types into a bitmask, then rebuilds the chain in a
-hard-coded priority: ngram types first, then draft-simple, then MTP.
-So `draft-mtp,ngram-mod` and `ngram-mod,draft-mtp` produce the same
-internal order (`ngram-mod` then `draft-mtp`) and the same runtimes.
-Verified: both orders decode ~25.3 t/s at 8k.
-
-### Results (decode generation t/s, 200 tokens, prose prompt)
-
-| ctx | MTP only | MTP + ngram-mod | verdict |
-|-----|----------|-----------------|---------|
-| 8k | 29.42 | 25.27 | ngram hurts -4.15 |
-| 16k | 26.42 | 21.13 | ngram hurts -5.29 |
-| 64k | 19.72 | 18.00 | ngram hurts -1.72 |
-
-At every depth, adding ngram-mod under MTP REDUCED decode throughput. The
-ngram draft costs its own forward pass / occupancy without adding enough
-accepted tokens on top of MTP. Baseline (no ngram) remains the production
-config (`--spec-type draft-mtp` only). Keep ngram-mod off.
-
-The historical harness was moved to `.AGENTTRASH/bench-scripts-20260901/` and is retained for audit only.
-it boots the v3 stack on a scratch port and reads authoritative
-`print_timing` lines. It requires the production service to be stopped.
-
-## See Also
-
-- [[frank-redo-2026-08-17]] — the fleet topologies this slot has served.
-- [[Shared Memory Architecture Dankclaw]] — unified memory sizing.
-- [[qwen3.8-strix-optimizations]] — llama.cpp ROCm0 baselines on this iGPU.
-- [[VLLM Overview]] — vLLM reference (retired for this slot).

serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md

@@ -205,7 +205,7 @@ answered 200. (Later the same day the pool DOUBLING to 401,408 positions for
 slot parity was COMMITTED (`2b12716`) and **deployed the same evening**: the
 boot now reserves `401408 positions (about 11.0 GiB)` and the engine-side total
 is `79.6 GiB in all` (62.9 weights + 11.0 KV + 5.7 working),
-see [[frank-dragoon-flash-server-v2-native]].)
+see [[frank-dragoon]].)
 
 **The NPU came back with the flip.** `amdxdna` bound to `0000:c6:00.1` (Strix/
 Krackan/Strix Halo NPU rev 11), firmware `amdnpu/17f0_11/npu_7.sbin`, `Initialized
@@ -229,8 +229,8 @@ lives in `~/.local/bin`.
 ## Related
 
 - [[frank-npu]] — the endpoint this retired, including the err=-19 root cause
-- [[halogen-flash-server]] — the runtime bought by the prefill figure
+- [[frank-dragoon]] — the runtime bought by the prefill figure
-- [[kingjones-strix-rocmfp4]] — the runtime it replaced (rollback target)
+- [[frank-dragoon]] — the runtime it replaced (rollback target)
 
 ## Citations
 

serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md

@@ -166,7 +166,7 @@ Not advisable.
 | Runtime | gfx1201 | Architectures | Note |
 |---|---|---|---|
 | **Affinity** | yes, built here | **V4-Flash only** — "the geometry is compiled in; there is no architecture abstraction" | 110 tok/s for 284B. Best engine for this box by an order of magnitude, wrong model |
-| **tcclaviger vLLM fork** | **yes — proven 2026-09-29** | qwen4exp measured 37.4-40.6 tok/s decode (see [[dankserver-qwen38-fp8-expert-offload-tuning]]); **`glm5next` registered** (`Glm5NextForCausalLM`/`MTP`, backend-split `amd/`/`common/`/`nvidia/`, `sparse_indexer.py` for DSA); r4d.so has **native mxfp4 + dsfp4 MoE entries** (`has_dsfp4() == True`); W4A16 MoE class (`R4dW4a16MoEExperts`) wired into the same tiered offload; TunableOp silently skipped under offload (needs `CLAV_TUNABLEOP_WITH_OFFLOAD=1`, and then corrupts numerics — do not use) | the ad hoc qwen-fp8 container's runtime; **the old "MXFP4 = emulation/missing kernels" verdict predates this fork and is dead** — the GLM-5.3-Flash gate is now the payload format + a boot probe, not kernels |
+| **tcclaviger vLLM fork** | **yes — proven 2026-09-29** | qwen4exp measured 37.4-40.6 tok/s decode (see [[frank-archon]]); **`glm5next` registered** (`Glm5NextForCausalLM`/`MTP`, backend-split `amd/`/`common/`/`nvidia/`, `sparse_indexer.py` for DSA); r4d.so has **native mxfp4 + dsfp4 MoE entries** (`has_dsfp4() == True`); W4A16 MoE class (`R4dW4a16MoEExperts`) wired into the same tiered offload; TunableOp silently skipped under offload (needs `CLAV_TUNABLEOP_WITH_OFFLOAD=1`, and then corrupts numerics — do not use) | the ad hoc qwen-fp8 container's runtime; **the old "MXFP4 = emulation/missing kernels" verdict predates this fork and is dead** — the GLM-5.3-Flash gate is now the payload format + a boot probe, not kernels |
 | **Lucebox** | yes, image on host | Qwen 3.8, DS4; **V4.1 on the unmerged `feat/ds41-engine` branch** | The only V4.1 path. **Refuses layer split and layer-major prefill for V4.1** |
 | **radiance-vLLM** (drwolfen) | yes — served frank-archon 2026-09-27 → 09-30 | welded to `qwen4_exp`; PLE / QSA / packed-HC live inside that model class | 12–20 person-days per architecture |
 | **FreeToken-ROCm** | partial | block-FP8 safetensors only (modelopt); known `CpuMoeExecutor` issue under ROCm | cannot load a GGUF |
@@ -221,7 +221,7 @@ gfx1201+gfx1201. hip:0 takes dense weights and the hot expert stack;
 
 ## See Also
 
-- [[frank-archon-lucebox-ds4]] — the retired Lucebox frank-archon runbook
+- [[frank-archon]] — the retired Lucebox frank-archon runbook
 - [[Dual-R9700-Cooperation]] — the PCIe/atomics facts behind the board swap
 - [[Dankserver System Specs]] — the host (RAM corrected 2026-09-28)
-- [[frank-archon-r9v-radiance-qwen38-q8]] — the ACTIVE frank-archon
+- [[frank-archon]] — the ACTIVE frank-archon

serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md

@@ -1,724 +0,0 @@
----
-type: research
-title: Qwen3.8-Flash-Next FP8 on tcclaviger — expert-offload decode tuning
-description: VERIFIED 2026-09-29 by measurement. The flag-level decode-tuning campaign on the ad hoc qwen-fp8 container (orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8, tcclaviger RDNA4 vLLM fork, 2x R9700): the >35 tok/s target is MET at 37.4-40.6 tok/s by turning prefetch OFF — telemetry measured its precision at 36%, wasting ~half of all gather traffic — while the original 26.8 tok/s incumbent (prefetch on) survived every other challenger. Residual upside is the fork's gather read path (effective ~6.5-9.5 GB/s vs 22.7 GB/s the link delivers at the same slab size).
-tags:
-  - dankserver
-  - r9700
-  - gfx1201
-  - qwen3.8
-  - fp8
-  - expert-offload
-  - speculative-decoding
-  - tuning
-timestamp: 2026-09-29
----
-
-# Qwen3.8-Flash-Next FP8 — expert-offload decode tuning
-
-> **Question this page answers.** On the tcclaviger RDNA4 vLLM fork serving
-> `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` across both R9700s, what
-> runtime flags actually move decode at a 32k working point?
->
-> **Answer in one line.** **37.44–40.57 tok/s — the >35 target is met — by
-> deleting `--expert-prefetch 1`.** Expert-cache telemetry measured the
-> prefetcher at **36% precision** (36,720 slabs prefetched, 13,369 served):
-> ~half of all gather traffic was useless copies, and the copies it "saved"
-> would have been installs anyway. The prefetch-on incumbent (26.8 tok/s)
-> survived every other challenger measured: gather geometry, spec depth,
-> TunableOp, scheduler batch knobs, concurrency, ngram-only drafting.
-
-This is the ad hoc `qwen-fp8` container on Dankserver (rootless podman,
-image `docker.io/tcclaviger/vllm:latest`, engine
-`v0.29.0.dev0+g2bdbbc8080`), NOT the `frank-archon` systemd service —
-the R9V production unit is untouched, so no stop-and-mask was required for
-any of this. All numbers are decode tok/s measured from the server's own
-`usage.completion_tokens / (wall - ttft)` — SSE chunk counting is invalid
-under speculative decoding because rejected drafts change the accounting.
-
-## Deployment shape (the tuned incumbent)
-
-Serve script on the host (`/home/dank/serve_qwen_fp8.sh`), env hooks in
-CAPS. **PREFETCH must be unset** (omitting the flag; the fork's default is
-off):
-
-```
-MODEL=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8   (~172.8 GiB, 131 shards)
-IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
-# image ID 3322dbbbc96d; local alias qwen-fp8-runtime:tuned-20260929;
-# the floating :latest tag moved once already — always launch by digest.
-# repo tag of this state: qwen-fp8-tuned-20260929
---tensor-parallel-size 2  --max-model-len 32768  --gpu-memory-utilization 0.98
---kv-cache-memory-bytes 2147483648  --max-num-seqs 4  --enable-prefix-caching
---speculative-config '{"method":"mtp","num_speculative_tokens":4}'
-# NO --expert-prefetch: measured 36% precision, ~50% wasted gather traffic
-```
-
-Published to the host LAN via the serve script's `PORTMAP` hook
-(`PORTMAP=7773:7770`): health and completions answer on
-`http://192.168.0.235:7773` (verified host-side 2026-09-29; 256k config,
-decode 36.49/34.55/35.98 through the published path). Host port 7773 was
-chosen over 7770 because `frank-archon` (7770) is stopped but ENABLED —
-any future start would collide with a published 7770.
-
-Device notes carried from the bring-up: `--device /dev/dri` must be the
-WHOLE directory (individual renderD nodes enumerate but refuse contexts —
-"No CUDA GPUs are available" while device_count() reports 2); the image
-entrypoint execs `vllm serve` itself.
-
-The memory plan this produces (from the engine's own `expert_plan.py` log):
-
-| region | size |
-|---|---:|
-| expert cache space | 15.05 GiB = 6,574 slots x 2.344 MiB |
-|   VRAM-only resident layers | 3.52 GiB (layers 0-2, 1,536 experts) |
-|   swap region | 11.53 GiB (5,038 slots, 111-112/layer) |
-| host-backed experts | 52.74 GiB per rank, 23,040 experts |
-| KV reserve (one max-len request +8%) | 0.36 GiB of the 2 GiB pool |
-| CUDA graphs | 1.93 GiB (21 graphs; 1.32 GiB at `--max-num-seqs 2`) |
-| runtime overhead | 7.6 GiB (non-torch + buffers + activation) |
-
-## Measured arms (2026-09-29)
-
-Decode tok/s at 166 / 1087 / 4064 prompt tokens, median of 3 runs after a
-64-token warm pass, temperature 0, max_tokens 256:
-
-| Arm | 166 | 1087 | 4064 | Acceptance | Verdict |
-|---|---:|---:|---:|---:|---|
-| **MTP4, stock 8x16, PREFETCH OFF** | **40.57** | **37.44** | **37.57** | 54.0% | **NEW INCUMBENT — target met** |
-| MTP4, stock 8x16, prefetch ON | 26.80 | 25.15 | 26.81 | 60.7% | old incumbent (+51%/+40% worse) |
-| MTP4, prefetch ON (re-check) | 25.80 | 24.38 | 26.63 | 47.9% | reproduces (noise band ±1) |
-| MTP4, prefetch ON, gather 16x32 | 24.30 | 22.77 | 24.11 | 50.3% | −9% — ruled out |
-| MTP4, prefetch ON, gather 32x64 | 26.58 | 24.94 | 26.34 | 49.6% | parity — ruled out |
-| MTP3, prefetch ON | 24.47 | 24.41 | 24.44 | 66.7% | −9%, flat in ctx — ruled out |
-| MTP6, prefetch ON | 19.07 | 17.81 | 13.54 | 17.4% | collapses; crosses UVA threshold — ruled out |
-| MTP4, prefetch ON, MNBT 8192 | 29.32 | 24.12 | 23.65 | — | neutral — ruled out |
-| MTP4, prefetch ON, PREFETCH depth 2 | 25.83 | 25.06 | 25.87 | 56.0% | neutral — ruled out |
-| MTP4, prefetch ON, TunableOp w/ offload | 23.59 | (79.82*) | (79.89*) | (96.5%*) | **rejected — corrupts** |
-| 2-way concurrency, prefetch ON (2 streams) | — | — | 15.06 + 23.11 = **25.49 summed** | — | no gain vs 26.8 single |
-| ngram-only drafter | — | — | — | — | fork bug: PLE inputs unprepared |
-| 256k ctx (MAXLEN=262144, KV 4.5 / 4.03 GiB, prefetch off) | 36.65 / 36.48 | 34.21 / 34.13 | 34.77 / **35.02** | 53.0 / 39.4% | boots and serves; band 34.1-36.5 straddles 35 within noise; arena identical (5,648) at both KV sizes |
-
-\* The TunableOp numbers are invalid by construction: the tuned GEMMs
-changed generated behavior (repetition-loop, the server's
-"repetition detected" halt firing at ~140-146 tokens instead of the full
-256, acceptance jumping 60.7% → 96.5%), the 79.8 tok/s figures were
-measured on those halted degenerate completions, and the text-only check on
-a fresh prompt came back 20.01 tok/s — *below* baseline. GEMM autotuning
-that changes output text is broken tuning.
-
-The decode curve without spec decode is dead flat 12.11 tok/s from 304 to
-16,095 tokens — decode is context-independent on this shape, bound per-step
-by the expert gather, not by KV depth.
-
-## Telemetry — the measurement that found the win
-
-`R4D_LRU_TELEMETRY=1` arms per-layer counters; a torch-profiler window
-(`/start_profile` → traffic → `/stop_profile`) makes the fork log and write
-`expert_cache_rank<N>.json` beside the traces. **`torch_profiler_with_stack=false`
-is mandatory in the profiler config** — the default (true) grew a 6.5-min
-window's trace buffer ~40 GB and the kernel OOM-killed the engine core
-(anon-rss 95.2 GB, `oom-kill` in `journalctl -k`). The short window at
-166 ctx (~15 s of traffic) completed cleanly with `with_stack=false`.
-
-The fork's own numbers (102 steps, both ranks identical):
-
-```
-hit rate 0.8245 (137,084/166,262)   host installs 15,814
-read-through 13,364                 prefetched 36,720
-served by prefetch (migrations) 13,369
-```
-
-- **The LRU is good**: 82.45% hit rate overall; layers 0-2 pinned at
-  100%; swap-region layers 78-81%. The "arena holds only 27% of experts
-  → 73% miss" arithmetic was wrong — temporal reuse carries it.
-- **The prefetcher is the poison**: 36,720 slabs moved to serve 13,369
-  (36% precision). The 23,351 unused copies are ~0.7 GB/step/rank of pure
-  waste — roughly half of all gather traffic — and the 13,369 "savings"
-  would have been installs anyway. Depth-1 prefetch also cannot cover the
-  ~2,400-slab per-step working set with its 16 slots/layer (768 total).
-- Per-layer-step at a swap layer: ~30.6 routed, 24.8 hits, 3.3 installs,
-  2.6 read-through, 8.0 prefetched of which 1.7-3.3 useful.
-
-## The wall, measured
-
-- **PCIe link (pcie-link-probe):** GPUs `03:00.0`/`07:00.0` LnkCap/LnkSta
-  32GT/s x16; root ports `00:01.1`/`00:01.3` cap x8. Effective **Gen5 x8 =
-  31.5 GB/s/direction**. `rocm-smi pcie_link_speed: 25` is the ASPM idle
-  downshift, not the loaded rate.
-- **H2D (HIP probe, pinned):** 2.345 MiB (the engine's exact expert slab)
-  **22.74 GB/s**; 16 MiB 28.19; 1 GiB 28.74 (91% of ceiling). Pinned ≈
-  pageable — staging is not the lever. Single-transfer latency at slab size
-  105.7 µs vs 78 µs ideal → ~21% per-transfer overhead.
-- **Engine's effective gather rate:** ~6.5-9.5 GB/s = **30-42% of what the
-  link delivers at the same size**. The bottleneck is the fork's read path
-  (chunk/lane geometry, per-transfer cost), not the wire.
-- **Duty cycle:** peak GPU util 100%, smooth ~29% with prefetch on — each
-  step is ~100 ms of compute then hundreds of ms waiting on expert
-  streaming. rocm-smi reports *utilization*, which reads ~0 while weights
-  load during an arm's reload; VRAM is the honest panel there.
-
-## What the remaining flags cannot do (all measured)
-
-- **Gather geometry** — 16x32 and 32x64 measured no better despite the
-  fork's "tuned at the Gen4 ceiling" comment; the constants are already at
-  parity on Gen5 x8. The read-path gap is structural, not a tuning knob.
-- **Spec depth** — 4 is the optimum both ways (with prefetch on). Depth 6
-  additionally proves the UVA read-through path (`step tokens >
-  R4D_LRU_UVA_TOKENS=5`) is *slower* than copy-in.
-- **Prefetch anything** — depth 2 neutral, depth 1 the new negative; the
-  tier is 3x too small for the working set and the predictor is 36%
-  precise. `expert_prefetch_k/budget/slots` exist (defaults 20/14/16) but
-  the precision result makes tuning them pointless.
-- **Arena growth via shrink** — `UTIL=0.99` fails boot (`Free memory on
-  device cuda:1 (31.29/31.86)`); `--max-num-seqs 2` fails CUDA-graph
-  capture (`Constraints violated (L['query_start_loc'].size()[0])`); and
-  the MNBT 8192 arm already proved a ~1 GiB arena shift is decode-neutral
-  in both directions.
-- **Concurrency** — under TP=2 both streams' tokens are computed on both
-  cards, so each rank gathers the union of both streams' misses; the links
-  do not add capacity and the shared arena thrashes. Summed 2-way (25.49)
-  measured *below* single-stream (26.8) with prefetch on.
-- **ngram-only drafting** — fails at engine init:
-  `RuntimeError: PLE inputs were not prepared`; the fork prepares this
-  model's PLE inputs only on the MTP spec path. Needs a fork patch, not a
-  config.
-- **V-only q8 KV** — stock `--kv-cache-dtype` offers fp8 variants only
-  (K+V together); a V-only q8 needs a fork patch.
-
-## Fork bugs / traps found (for the engine authors or a future patch)
-
-1. `CLAV_TUNABLEOP_SWEEP=1` is silently skipped whenever expert offload is
-   active unless `CLAV_TUNABLEOP_WITH_OFFLOAD=1` is also set
-   (`tunableop_warmup.py`).
-2. TunableOp-with-offload, once forced on: +6.7 min warmup, no persistent
-   cache (container-local), and corrupt numerics (repetition loops,
-   early-EOS halts, invalid measurements). Rejected outright.
-3. The torch-profiler stop path hangs the workers on ROCm when
-   `torch_profiler_with_stack` is true (default): >10 min hangs on two
-   attempts while the AsyncLLM wrapper reports "stopped successfully" in
-   1 s; the default stack-tracing grew a 6.5-min window's buffer ~40 GB
-   and OOM-killed the engine core. With `with_stack=false` the stop
-   completes and the telemetry report runs. The telemetry counters remain
-   reachable ONLY through that profiler stop — no direct RPC trigger
-   exists (`telemetry_report` has exactly one caller, `gpu_worker.py`).
-4. ngram spec method does not prepare PLE inputs (above).
-
-## Where the headroom lives now
-
-The 35 tok/s target is met (37.4-40.6) with prefetch off. What is left:
-
-1. **Fork patch to the gather read path** — the ~6.5-9.5 → 22.7 GB/s
-   software gap at slab size. If the gather reached wire rate, decode
-   lands in the 60+ tok/s band before acceptance effects. Code work in
-   `expert_lru.py`'s gather (`R4D_LRU_GATHER_*` is read at import; the
-   copy-in path itself is compiled — see `/app/r4dhip/r4d.so`).
-2. **256k ctx — measured, final**: boots at `MAXLEN=262144` with
-   `KV_BYTES=4831838208` or `4328521728` alike — a 262,144-token window
-   needs **4.0 GiB** KV (hybrid attention + mamba pages ≈ 16 KiB/token,
-   not the 32k plan's 9,190 B/token; 3 GiB boot-refuses and the engine
-   estimates max len 184,224 at that pool; the exact-4.0 override loses
-   ~20 MB to accounting and boots as 3.98 < 4.0). Decode band
-   **34.1-36.5** across both KV sizes — the arena is **identical (5,648
-   slots)** at 4.5 and 4.03 GiB (the planner buckets; the trim went to
-   margin, not slots), so the window costs ~10% decode vs the 32k config
-   and no KV trim recovers it. The alternative that keeps the full
-   37.4-40.6 band is `MAXLEN=184224` at 3 GiB KV. Correction: the planner
-   sizes the arena AFTER the KV override (the earlier "before" reading was
-   wrong) — the arena is what remains.
-3. **2+ slots at long ctx** — the KV pool post-spec holds ~66k tokens at
-   2 GiB (~2x 32k streams); concurrency summed *below* parity at 4064 ctx,
-   so slots are a capacity feature, not a throughput one. Two full 256k
-   streams need ~8-10 GiB of KV — that fits the arithmetic only by cutting
-   the arena below usefulness; slots-at-256k is a fork-patch or
-   smaller-payload question.
-
-## Reproducing
-
-Host-side scratch scripts (2026-09-29): `/home/dank/serve_qwen_fp8.sh` (the
-serve script with all hooks), `/home/dank/mtp_measure.py` (usage-counter
-decode measure, 3 depths), `/home/dank/mtp_measure_short.py` (single depth,
-for profile windows), `/home/dank/bench2way.py` (concurrency), plus the arm
-scripts `arm_*.sh` / `sweep_*.sh` and the detached runner `/home/dank/job.sh`
-(outputs under `/home/dank/jobs/NAME.out|.rc|.done`). The measure scripts
-are `podman cp`'d into the container after every recreate (the serve script
-re-copies them; a recreate wipes container-local files).
-
-## 2026-10-01 — production cutovers (qwen FP8+MTP4, then bf16 no-spec)
-
-Two same-day cutovers moved this page's findings into production:
-
-1. **Qwen FP8 + MTP4 replaced the failed GLM/DFlash2 experiment** (the GLM
-   window is documented in [[frank-archon-glm53-flash-w4a16]]; git tag
-   `frank-archon-fp8-mtp-20261001` is the last commit of that shape). The
-   production trio took MAXLEN 200000 x 3, fp8 KV, MTP4, prefetch off —
-   measured 37.4-40.6 tok/s at ~54% acceptance.
-2. **The overnight redo campaign re-measured EVERY lever at matched
-   geometry** (prefetch OFF everywhere, MAXLEN 200000/SEQS 3/fp8 baseline)
-   and concluded the day: the incumbent cutover to **bf16 KV, no
-   speculative decode** (Frank's call).
-
-### What the redo campaign measured (all at shallow depth, 166/1087/4064)
-
-| lever | verdict |
-|---|---|
-| prefetch (re-confirmed) | the whole night's +40-51% lives in deleting it; the old 12.11 "no-spec baseline" was itself prefetch-taxed |
-| gather geometry | **R4D_LRU_GATHER_CHUNKS=32 / LANES=64 = +18.7%** (49.25 @4064), the win GROWS with depth (39.5@166 -> 54.1@1087); g8x16 mild, g64x128 past peak |
-| KV dtype | bf16 (`auto`) +4% decode on fp8-era numbers but halves attention-KV capacity (measured 14,047 vs 7,926 B/token; mamba states unaffected) |
-| spec family | no-spec fastest once the tax was gone (41.46/41.29/41.47); MTP4's drafter reserves ~2.6 GiB/rank beside the pool; untaxed MTP re-measures never completed |
-| MNBT | 512 doubles TTFT (2.46 s vs 1.36 s @4064); 1024-4096 interchangeable; 12288 -2.5% + arena; **locked 1024** |
-| UVA | 2048 POISON (-39%); 8192 ~parity |
-| arena | flat ±1.4% from ~6,830 down to ~5,500 slots; knee below |
-| six slots | every SEQS-6 KV sizing refused at engine init |
-
-Depth caveat: every number above is at a near-empty KV. Deep-context
-behavior (the gather win growing with depth, bf16's decode cost at depth)
-is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
-
-### The deployed shape (2026-10-03 shape G: 3 slots x 200704, fp8, precise memory, MNBT 2048, arena at the flag ceiling)
-
-```
-FRANK_ARCHON_MAXLEN=200704   200704-token window
-FRANK_ARCHON_SEQS=3          --max-num-seqs 3 (SEQS=2 CANNOT boot - below)
-FRANK_ARCHON_KV_BYTES=3166500000   PER RANK (fp8, 7,892 B/token -> 400,700 tokens)
-FRANK_ARCHON_KV_DTYPE=fp8    --kv-cache-dtype fp8 (quantized)
-FRANK_ARCHON_MNBT=2048       --max-num-batched-tokens 2048 (arena-free, see shape F)
-FRANK_ARCHON_UTIL=0.982      budget = total x util, the law (1.0 unbootable, see below)
-FRANK_ARCHON_CUDAGRAPH_BASE_MB=0 / _PER_GRAPH_MB=12  -e VLLM_CUDAGRAPH_MEM_*
-FRANK_ARCHON_PRECISE_MEMORY=1      --expert-precise-memory (overhead 7.60 -> 5.25 GiB)
-FRANK_ARCHON_EXPERT_OFFLOAD_MEM=120  --expert-offload-mem (host-region cap, explicit)
-FRANK_ARCHON_GPU_MAX_HW_QUEUES=2   -e GPU_MAX_HW_QUEUES=2 (HSA queues/GPU)
-FRANK_ARCHON_CACHE_DIR=/home/dank/.cache/frank-archon   mounted at /root/.cache
-R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64  (container env)
-NO --speculative-config, NO --expert-prefetch
-FRANK_ARCHON_NO_PREFIX_CACHING=1  --no-enable-prefix-caching (2026-10-05, the QFN align-mode GDN state fault — below)
-```
-
-### 2026-10-05 — prefix caching OFF: the fork's QFN align-mode GDN state fault
-
-**Symptom (Frank, 2026-10-04):** output turned repetitive/corrupted at
-~40k tokens into a long conversation; short sessions stayed clean.
-
-**Evidence (live `podman logs frank-archon-qwen`, the 2026-10-04 18:19
-boot):** one 0.6 s burst at 18:46:07 PDT — 5.6 h into service, one
-running request, and the engine's own 18:46:08 log line reading
-"Prefix cache hit rate: 48.1%". 48 `gdn_hip` faults, then the
-process-lifetime silence cap:
-
-```
-[gdn_hip] ERROR gdn_decode_r: state out of ssm-cache range: max|S|=0
-  with non-finite values (slot=42 head=3 limit=3.40282e+38) fault #1-23
-[gdn_hip] ERROR gdn_decode_r: ... (slot=42 head=8 ...) fault #24-46
-[gdn_hip] ERROR: state fault report limit (32) reached; further faults
-  are counted silently
-```
-
-One physical SSM slot (42), two heads (3 and 8 — one per TP rank), 23
-GDN layers each, the state non-finite (NaN). The guard's own text in
-this build: *"known Qwen3.8-Flash-Next (QFN) flaw under prefix caching
-and is being actively worked; output quality for this request may
-degrade. Mitigation: --no-enable-prefix-caching."*
-
-**Mechanism.** With prefix caching on, the fork defaults QFN to
-`mamba_cache_mode=align` (boot log: "Mamba cache mode is set to 'align'
-for Qwen4ExpForConditionalGeneration … when prefix caching is enabled";
-`config/cache.py`: "none" is the no-prefix-cache mode, "all" is
-deprecated — align is the only sound mode WITH the flag). Align mode
-checkpoints each sequence's GDN (Gated DeltaNet) recurrent state into a
-mamba page per block and RE-SEEDS the state from that page when a
-request resumes from a cached prefix. The block size is raised to
-**3136 tokens** for the page-size equality (boot log: "Setting
-attention block size to 3136 tokens to ensure that attention page size
-is >= mamba page size" + "Padding mamba page size by 0.13% … exactly
-equal"). The corrupt state landed at the **13th boundary: 13 × 3136 =
-40,768 tokens** — the reported "~40k" mark. The NaN persisted in the
-mamba page, so every later turn of that conversation re-seeded the same
-non-finite state for those two heads (silently — the fault-report cap
-was already spent) while fresh short sessions looked fine.
-
-**RAM ruled out the same day** (the 2026-10-04 17:17 host reboot is when
-the current memory speed took effect): zero MCE/EDAC kernel records
-since boot; an 8 GiB in-band pattern check stable over 60 s under live
-load; 4800 MT/s is the JEDEC DDR5 floor on 5600 MT/s-rated DIMMs (not an
-overclock); and a two-head, single-slot NaN state is a state-path fault,
-not scattered finite bit corruption.
-
-**Fix.** `FRANK_ARCHON_NO_PREFIX_CACHING=1` → `--no-enable-prefix-caching`
-(the fork-named mitigation; the flag renders in both directions from the
-knob so a fork default change cannot silently flip the contract). Cost:
-no prefix reuse — every turn re-prefills the whole history instead of at
-most 3135 tail tokens. A silently corrupted conversation is worse than a
-slower first token.
-
-**Verified 2026-10-05 (deploy b820d34):**
-
-- Engine argv carries `--no-enable-prefix-caching`; health 200; warm boot
-  `init engine … took 76.57 s` (the mounted compile cache is intact).
-- Boot plan unchanged, as predicted: expert cache **17.77 GiB / 7,762
-  slots** (the flag touches no plan term), CUDA-graph estimate 0.07 GiB
-  vs 0.08 GiB captured (inside the 12 MiB/graph charge).
-- `GPU KV cache size: 424,678 tokens` — **up** from 401,408 at the same
-  3,166,500,000 B ask: the pool now prices ~7,456 B/token (was 7,891.8),
-  the mamba/side per-token term dropping once the per-block checkpoint
-  pages are gone; the 3136 attention block-size raise nonetheless stays
-  (the page-size invariant is enforced in mode "none" too). Frank's
-  400,700-token floor is cleared with ~24k of headroom.
-- **Zero `gdn_hip` lines** in the boot-and-traffic window (only the two
-  INFO module banners mention the name).
-- Boundary probe (single request, no prefix reuse): prompt 39,280 +
-  completion 1,491 = **40,771 total tokens — across the 40,768 line**;
-  the task (write 1–400, one per line) completed coherently: 399 integers
-  parsed, 0.997 monotone, tail `… 398 399 400`. Pre-fix, this boundary is
-  exactly where the conversation degraded (the 13th 3136-page re-seed).
-- e2e: 32/33 pass; the one failure is the admin spec's
-  192.168.0.112:8989 leg with FrankBook-Pro asleep (environmental,
-  unrelated to this change; re-run `tests/e2e/run.sh` once the mini is up).
-
-**RESULT — REVERTED THE SAME DAY.** With every mechanical check green
-(zero faults, coherent 40,771-token probe), live long conversations
-STILL degraded at ~40k on the running endpoint (Frank, direct
-observation). The container serving that window (`--no-enable-prefix-
-caching`, booted 2026-10-05 12:47 PDT) logged ZERO `gdn_hip` faults
-while the corruption reproduced — so this recurrence is **fault-free**:
-the NaN-state guard did not fire, which the re-seed theory requires.
-Open hypotheses, in order: (1) the poisoned transcript is being re-fed —
-corrupted assistant turns already in the history keep producing
-corrupted continuations even with perfect state, so the same
-conversation cannot distinguish fixed from unfixed; (2) a second
-long-context mechanism with no guard coverage — fp8 KV degradation at
-depth is the next suspect (a bf16-KV boot costs ~2× the pool per token
-and would settle it). Prefix caching is **back ON**
-(`FRANK_ARCHON_NO_PREFIX_CACHING` commented out — the cost bought no
-fix). The knob and the serve branch stay for a future fork build with a
-fixed GDN path. Next diagnostic: reproduce on a FRESH conversation; if
-it still breaks at ~40k with zero faults, A/B the KV dtype.
-
-**RESOLVED (2026-10-05 evening): it was the RAM.** Frank dropped the
-memory from 4800 to **4600 MT/s** (in effect since the 2026-10-05 12:21
-PDT host boot) and the corruption stopped: zero `gdn_hip` ERROR lines in
-the container since, zero MCE records, and the live agent session on
-this endpoint past 140k context, "comfy cozy" (Frank). Marginal DDR5 on
-four DIMMs is a common cause for BOTH eras: the 2026-10-04 NaN burst
-(corrupted weight/activation traffic poisoning a GDN state write — the
-guard-visible path) and the fault-free ~40k degradation (silent flips in
-KV/expert data, which no guard covers on non-ECC memory). The re-seed
-theory is therefore UNPROVEN, not disproven — the entire caching
-experiment ran on marginal RAM and was confounded on both arms.
-Post-mortem on the rule-out: consumer non-ECC RAM raises no MCE for a
-silent flip; a 60 s / 8 GiB pattern sample covers ~0.5 % of 246 GiB;
-"4800 is the JEDEC floor on 5600-rated DIMMs" is a rating, not a
-stability proof at four-DIMM loading. The revert test was the decisive
-instrument and it pointed at the RAM all along. Prefix caching stays ON
-(zero faults under heavy re-send traffic since the speed drop); the knob
-stays for a future fork fix. The fp8-KV A/B is NOT needed.
-
-### Shape F/G (2026-10-03): where the free VRAM actually is, and what forcing it cost
-
-`rocm-smi` showed **3.57 GiB free per R9700** at the deployed shape E (28.28 GiB
-used of 31.86 GiB, after hours of real traffic — so it is the high-water mark,
-not an idle artifact). Growing the expert arena into it is NOT a matter of
-raising `--gpu-memory-utilization`, because the arena is
-
-```
-arena = budget - non-expert - KV reserve - CUDA graphs - runtime overhead
-budget = device_total x gpu_memory_utilization          # expert_plan.py:567
-```
-
-and every term on the right is either fixed or a lie:
-
-| term | plan charges | measured reality | reachable from our side? |
-|---|---|---|---|
-| `budget` | 31.22 GiB (0.98) | field is `le=1` so 31.86 GiB is the field max, but 1.0 is UNBOOTABLE (below) | **yes** — shipped 0.982 |
-| `non-expert weights` | 6.28 GiB | checkpoint census | no |
-| `KV reserve` | 1.91 GiB (registry 9,190 B/token x 200704 +8%) | real pool bills 2.93 GiB at 7,892 B/token | no (registry constant) |
-| `CUDA graphs` | 0.55 GiB = 256 MiB + 52 MiB x 6 | `Graph capturing finished in 3 secs, took 0.31 GiB` | **yes** — `VLLM_CUDAGRAPH_MEM_BASE_MB` / `_PER_GRAPH_MB`; `VLLM_CUDAGRAPH_MEM_ESTIMATE_MB` pins the whole term outright (checked first, pair ignored when set) |
-| `runtime overhead` | 5.25 GiB (`--expert-precise-memory`) | far lower at idle | no (registry constant) |
-
-Shape F took the graph term (0.55 -> 0.32 GiB) and nudged util to 0.982:
-**arena 17.52 GiB / 7,653 slots measured**, +131 slots. Shape G takes the
-graph term further (0.32 -> 0.07 GiB) at the same util.
-
-**Shape G takes the graph term to the measured floor, util to the law.** The
-graph base pool is the actual over-charge: shape F's own boot reported
-`Graph capturing finished in 4 secs, took -0.06 GiB` — a **negative** capture
-delta, so the 256 MiB base was fiction. Charging `0 base + 12 MiB/graph`
-(0.07 GiB for the shipped 6-graph plan, and still per-graph so an 18-graph
-spec-decode shape pays 0.21 GiB) lands the plan at budget 31.29 / graphs 0.07 /
-**arena 17.77 GiB / 7,762 slots measured**. The KV ask went to
-`3,166,500,000` B/rank, which the engine's own 7,891.8 B/token price lands at
-**401,408 tokens measured / 2.00x concurrency** — the first shape holding two
-full-length conversations with real spare. (The ask/landed delta is block
-alignment: 196 x 2048. The price is shape-dependent, 7,701 B/token at 262144.)
-
-**UTIL 1.00 is not a knife-edge on this fork, it is UNBOOTABLE — measured
-2026-10-03.** One deploy took the field maximum to buy the last 0.57 GiB of
-arena; both workers refused init BEFORE loading a single weight, twice:
-`ValueError: Free memory on device cuda:N (31.29/31.86 GiB ...)` — the worker
-demands free > budget and budget == total, so `le=1` on the CLI field is a
-trap: the engine dies in a 2-restart crash loop with zero payload progress.
-So `0.982` is a LAW, not a preference, and the "why 0.982" answer now has two
-proofs: this ValueError, and the historical ceiling deaths (the 2x262144 bf16
-boot at 0.13 GiB free; the 5x262144 boot at "0 bytes free" on a real
-995-token request). Everything beyond ~7,912 slots needs the registry edited
-inside the image (the fork's log line says exactly that: *"Update the
-registry entry if this drifts"*), not a deploy.
-
-**The host region was the last arena-plan term still autofitting per boot —
-now it is a literal (2026-10-04).** With no `--expert-offload-mem` the planner
-binds the host region to `MemAvailable` AT BOOT minus `PLE 47.68 - 2 x 7.0
-process - 5.0 api/engine - 5% margin`: the 2026-10-05 boot read 237.97 GiB and
-offered 159.39 GiB usable, while the same host reads **75.4 GiB available while
-running** — the bound tracks whatever the co-tenants hold at that moment. The
-demand is 105.48 GiB (52.74 GiB/rank x 2, 23,040 swap-target experts), and a
-boot where the bound drops near it silently converts next-hottest experts to
-VRAM-only (the `pinned_extra` warning is the only trace).
-`FRANK_ARCHON_EXPERT_OFFLOAD_MEM=120` pins it: measured demand + 14% headroom,
-on a host with 246.1 GiB total. Semantics verified in the shipped image
-(`offloader/expert_plan.py`): the cap binds as the bound, an exceedance only
-warns, and the plan rewrites the field to the cap it met (+1 MiB/rank).
-
-**MNBT 1024 -> 2048 is free on every axis the plan can see.** The registry
-overhead model is `base + 0.22 GiB x max(0, (nbt - 4096)/1024)` anchored at
-nbt 4096, so any rung up to 4096 charges the identical 5.25 GiB; the fitted 5.25
-was itself fitted at nbt 4096, which means 2048 is covered by an activation
-figure measured at a **larger** batch. SEQS 3 boots at both 1024 and 2048
-(measured 2026-10-02 — the `query_start_loc` `ConstraintViolationError` is a
-SEQS property). The costs are a one-time recompile (the compile **range** is
-part of the cache key, so `FRANK_ARCHON_CACHE_DIR` pays once) and a wider
-prefill chunk, which the 09-29 sweep found decode-interchangeable across
-1024-4096 while 512 doubled TTFT at 4064 ctx (2.46 s vs 1.36 s).
-
-**`--expert-precise-memory` is the arena lever that does not touch the window.**
-It attacks the planner's overhead term (shape F above attacks the graph term —
-the third and last term this repo can reach), and the flag swaps the default
-registry figure for the fitted one. Measured
-2026-10-02 by a boot probe on the shape-D base (production stopped, candidate
-booted AS the service; numbers are the engine's own plan lines):
-
-| plan line | shape D | **shape E** |
-|---|---|---|
-| `runtime overhead (non-torch + buffers + activation)` | 7.60 GiB | **5.25 GiB** (`precise (gfx1201 TP2 nbt4096/seqs16/len262144)`) |
-| `expert cache space` | 14.87 GiB | **17.22 GiB** |
-| slots | 6,496 | **7,522** (+1,026, +16%) |
-| `swap region` | - | 13.7 GiB / 5,986 slots, 133-134 per layer |
-| boot | - | healthy in 420 s, no OOM |
-
-`'expert_precise_memory': True` also appears in `non-default args`, so a silent
-argparse drop is visible in the boot log. The flag lands **in the arena, not the
-pool** - `--kv-cache-memory-bytes` is untouched at 3.15e9/rank (option A), because
-the pool is not an arena lever and cutting it further would only cost
-concurrency: a 2.2e9 pool was probed in the same window and reports 277,686
-tokens / **1.38x**, so it is the conservative fallback if free VRAM ever binds,
-not the shipped value.
-
-**The compile cache is a boot-cost fix, and it was worth 173 s per boot.**
-Measured 2026-10-02: the image ships **no `/root/.cache` at all** (a fresh
-container: `No such file or directory`), and after a boot that directory holds
-`vllm` at **1,154,714,333 B in 4,665 files** plus `comgr` at 229 MB, all inside
-the container's ephemeral layer. Every
-`podman rm -f` - i.e. every deploy, restart and reboot - discards it, and the
-engine re-pays:
-
-```
-Using cache directory: /root/.cache/vllm/torch_compile_cache/b265bd396a/rank_0_0/backbone
-Dynamo bytecode transform time: 4.99 s
-Compiling a graph for compile range (1, 1024) takes 173.19 s
-saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/…
-torch.compile took 181.20 s in total
-init engine (profile, create kv cache, warmup model) took 217.26 s (compilation: 181.20 s)
-```
-
-Mounting a host directory at `/root/.cache` makes every later boot load the AOT
-artifact instead. The mount is deliberately the **parent**, not
-`/root/.cache/vllm`: `VLLM_CACHE_ROOT` defaults to the `vllm` child, and
-`comgr` is the HIP-kernel JIT cache, so one mount persists both — and it hides
-nothing, because the image ships no `/root/.cache`. The cache key
-includes the compile range, so the MNBT choice costs exactly one compilation.
-Entries are version- and config-hashed: a stale entry is a miss, never a wrong
-answer.
-
-**Proven live on the deployed endpoint, 2026-10-02, consecutive boots of this
-exact shape:**
-
-| boot | engine log |
-|---|---|
-| cold (deploy) | `Compiling a graph for compile range (1, 1024) takes 179.24 s` → `torch.compile took 188.26 s in total` |
-| warm (restart) | `Directly load the compiled graph(s) for compile range (1, 1024) from the cache, took 2.143 s` → `torch.compile took 3.97 s in total` → `init engine (profile, create kv cache, warmup model) took 41.13 s (compilation: 3.98 s)` |
-
-The host directory held **1.4 GB in 4,707 files** after the cold boot. Time to
-`/health` is dominated by the 172.8 GiB payload load (the warm boot reached
-health in 490 s), so the win is the ~184 s of inductor/AOT build rather than a
-proportional cut in boot time.
-
-`GPU_MAX_HW_QUEUES=2` caps the HSA hardware queues per GPU. Verified a real knob
-rather than a no-op - the literal is read out of
-`/opt/rocm/core-10.0/lib/libhsa-runtime64.so` - and shipped deliberately
-alongside the 32x64 gather geometry. It is **not** A/B measured.
-
-**The pool, not the slot count, sets how many conversations live at once.**
-Measured live at this shape: `GPU KV cache size: 398,658 tokens, Maximum
-concurrency for 200,704 tokens per request: 1.99x` (3,146,129,408 B reserved).
-The per-token price is **shape-dependent**: the 262144 shape priced at 7,701
-B/token (6,100,000,000 B -> 792,069 tokens), this pool at 7,892 - ~2.5% dearer,
-so a concurrency figure never extrapolates across geometries. The endpoint
-advertises 3 slots: a third LONG request queues rather than being refused, and
-shorter requests use the spare blocks. Verified: three concurrent 67-token
-requests answered in parallel, ~4.1 s each. Two contexts at 200704 are a
-capacity contract, not a throughput one (2-way concurrency summed BELOW parity
-at 4064 ctx).
-
-#### `--max-num-seqs 2` is impossible on this model (measured 2026-10-02)
-
-Four single-variable boot probes against the then-deployed shape, production
-stopped for each:
-
-| MAXLEN | SEQS | MNBT | result |
-|---|---|---|---|
-| 262144 | 2 | 1024 | **Engine core initialization failed** |
-| 262144 | 2 | 2048 | **Engine core initialization failed** |
-| 262144 | 3 | 1024 | healthy - 792,069 tokens, 3.02x, arena 6,248 |
-| 262144 | 3 | 2048 | healthy - the then-deployed incumbent |
-
-The failure is `torch.fx.experimental.symbolic_shapes.ConstraintViolationError:
-Constraints violated (L['query_start_loc'].size()[0]) ... specialized it to be
-a constant (3)`, raised from AOT `build_guards(strict_error=True)` during
-`determine_available_memory -> profile_run -> _dummy_run`. `model_runner.py`
-sizes that input buffer `max_num_reqs + 1` and `_dummy_run` sets
-`num_reqs = min(max_num_batched_tokens, max_num_reqs)`, so the width is
-`SEQS + 1` for ANY `MNBT >= SEQS`: 3 at SEQS 2, 4 at SEQS 3. **MNBT is not a
-lever here** - it only moves the compile RANGE ((1,1024) vs (1,2048)), which
-is why the identical error appears at both rungs. `--enforce-eager` bypasses
-the path at the cost of CUDA graphs. The same failure class killed the
-2026-10-01 bf16 2 x 262144 shape.
-
-#### The arena lever is `max_model_len`, not `--kv-cache-memory-bytes`
-
-The fork's planner sizes expert cache as `budget - non-expert - KV reserve -
-graphs - overhead`, and the KV reserve is registry-priced at **9190 B/token x
-max_model_len** - not from the `--kv-cache-memory-bytes` value:
-
-| MAXLEN | KV reserve | arena |
-|---|---|---|
-| 262144 | 2.48 GiB | 14.30 GiB / **6,248 slots** |
-| 200704 | 1.91 GiB | 14.87 GiB / **6,496 slots** |
-
-D(reserve) 0.57 GiB = D(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9
-pool cut (2.75 GiB) bought **zero** arena slots. It buys runtime HEADROOM,
-which is still worth having - the shape it replaced died with `0 bytes free`
-on a real 995-token request. Grow the arena by lowering `max_model_len`, or by
-taking the overhead term with `--expert-precise-memory` (below); the pool is the
-headroom lever only.
-
-#### The other arena lever: `--expert-precise-memory` (overhead 7.60 -> 5.25 GiB)
-
-The flag is now ON. It was refused on 2026-10-01 for a reason that does not
-generalize: the 2 x 262144 bf16 shape was already at the ceiling (weights 6.28 +
-pool 6.80 + graphs 0.55 + arena 16.65 + non-torch ~1.45 = 31.73 of 31.86 GiB,
-NRestarts=3 at ~6.7 min each), so the 2.35 GiB the flag moves into the arena was
-the last stack over the line, not the cause. The plain bf16 2 x 262144 boot also
-did not stay up: 0 bytes free at runtime, and a real client request (995-token
-prompt, `max_tokens=261149`) killed the worker with `Tried to allocate 50.00 MiB
-... 0 bytes free`.
-
-At 200704 x 3 the flag fits with room and is worth **+1,026 slots (+16%)**. What
-made it affordable is the window trim, not a pool cut: the arena is
-DEVICE-resident, the pool is not part of the budget's arithmetic, and the 200704
-window's KV reserve (1.91 GiB) plus a smaller attention footprint is what the
-plan needed. The pool stays at 3.15e9/rank deliberately (option A) - cutting it
-further would cost concurrency (2.2e9 -> 1.38x) and buy no slots.
-
-Boot-probed 2026-10-02 (transient systemd units at the production uid +
-memlock, full-log capture; the rootless bench probe had masked the production
-memory environment). The night searched two concurrency contracts:
-
-| rung | result |
-|---|---|
-| 3 x 262144, bf16, KV 11,047,010,304 | **OOM at graph capture** — `Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB already allocated by PyTorch |
-| 3 x 245760, bf16 | **OOM identically** |
-| 3 x 237568, bf16 | PASS — KV 718,606 tokens |
-| 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 (the ceiling, not the flag) |
-| 2 x 262144, bf16 (no precise flag) | did not stay up either: 0 bytes free at runtime, worker killed by a real request |
-| 3 x 200704, fp8, pool sized for 2 contexts | **DEPLOYED 2026-10-02 (shape D)**: 398,658 tokens = 1.99x concurrency, arena 14.87 GiB / 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom |
-| **3 x 200704, fp8, + `--expert-precise-memory`** | shape E (2026-10-02): same 1.99x pool, overhead 5.25 GiB, arena 17.22 GiB / 7,522 slots, boot healthy in 420 s |
-| 3 x 200704, fp8, MNBT 2048, graph term 0.32 GiB, util 0.982 | shape F (2026-10-03): arena 17.52 GiB / **7,653 slots** measured, budget 31.29, `Estimated CUDA graph memory: 0.32 GiB`, KV unchanged 398,658 |
-| 3 x 200704, fp8, graph term 0.07 GiB, util **1.00**, KV 3.1665e9 | **DID NOT BOOT (2026-10-03)**: `ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` on both ranks, twice, zero weights loaded — the worker demands free > budget and budget == total. UTIL 1.0 is unbootable, full stop |
-| **3 x 200704, fp8, graph term 0.07 GiB, util 0.982, KV forced to 400,700** | **DEPLOYED 2026-10-03 (shape G)**: arena **17.77 GiB / 7,762 slots** measured (+109 over F, +240 over E), KV **401,408 tokens / 2.00x** concurrency measured, capture 0.31/0.10 GiB vs the 0.07 charge (under-charged, boots and holds), driver used 30.97 GiB -> **0.89 GiB real free per card** held through a live 4,096-token generation (~38-39 tok/s single-request, +13% vs the 33.99 shape-E baseline); rollback is UTIL or KV_BYTES down a notch |
-
-Six full-length contexts cannot fit at any KV dtype: the 6-seq graph/activation
-footprint leaves only 9.18 GiB for the pool, and 6 × 262144 needs 12.47 GB even
-at fp8.
-
-The old "KV_BYTES is PER RANK, 9/16 GiB OOM" law was the MTP4 WORKSPACE
-reservation, not a VRAM wall. Failures are clean and diagnostic: the engine
-reports `torch.OutOfMemoryError: Tried to allocate X GiB. GPU N has ... of
-which Y GiB is free`, and the fork's giant ERROR lines are split by journald —
-read them via `journalctl -o json`, not `-o cat` (and `podman logs`, which is
-only available while the container lives).
-
-Rollback to the 2026-09-29 fp8+MTP4 shape: git tag
-`frank-archon-fp8-mtp-20261001` (env trio + seam re-pinned there).
-
-## Parallelism options on this box (2026-10-01 survey)
-
-Surveyed from the running image's own CLI surface. Verdicts are specific to
-dankserver: 2x R9700 (gfx1201), a ~172.8 GiB expert-offload MoE, and a model
-with only 2 KV heads.
-
-| option | flag | verdict on this box |
-|---|---|---|
-| TP | `--tensor-parallel-size` | **DEPLOYED, =2.** All-reduce dispatch shows `R4D_AR` then `PYNCCL` |
-| DCP | `--decode-context-parallel-size` | **REJECTED at TP=2** — see below |
-| PP | `--pipeline-parallel-size` | supported; untried (stage handoff plus a TP-keyed expert plan) |
-| DP | `--data-parallel-size` (+`-rank`, `-start-rank`) | impossible: every replica holds a full model copy (172.8 GiB vs 31.86 GiB/card) |
-| EP | `--enable-expert-parallel` + `--all2all-backend` (+`--enable-eplb`, `--enable-dbo`, `--enable-elastic-ep`) | supported (`naive`, `allgather_reducescatter`, `deepep_high_throughput`, `deepep_low_latency`, `pplx`, `mori`, `nixl`); would replace the fork's tiered-offload LOCAL gather with a per-layer PCIe all-to-all |
-| multi-node | `--nnodes`, `--node-rank`, `--master-addr/port`, `--distributed-executor-backend` | supported; the actual path to more cards |
-
-### DCP is structurally unavailable here — do not re-test on 2 GPUs
-`--decode-context-parallel-size 2` fails `VllmConfig` validation at boot:
-
-```
-Decode context parallelism for GQA/MQA requires `--tensor-parallel-size` (2) to be
-greater than the model's total number of KV heads (2). Increase
-`--tensor-parallel-size` or set `--decode-context-parallel-size 1`.
-```
-
-DCP shards the KV along the *sequence* dimension after KV heads have been spread
-across the TP ranks, so it requires **TP > num_kv_heads**. This model has 2 KV
-heads (24 Q heads, GQA 12:1), so TP must be >= 4 — four GPUs, not two. Related
-fork facts: DCP *reuses* the TP ranks rather than multiplying them (it is not
-`TP x DCP`), and with PCP enabled the valid DCP sizes are `{1, pcp, tp x pcp}`
-(`vllm/config/parallel.py`). The KV memory a DCP win would have bought is real —
-each rank would hold half the sequence's KV — but it is unreachable without two
-more cards.
-
-### Measuring decode: the harness lives INSIDE the container
-`/home/dank/mtp_measure.py` imports `/bench_iter.py` by absolute path, so it must
-run in the engine's namespace and the files must be copied in first:
-
-```
-podman cp /home/dank/mtp_measure.py frank-archon-qwen:/
-podman cp /home/dank/bench_iter.py  frank-archon-qwen:/
-podman exec frank-archon-qwen python3 /mtp_measure.py
-```
-
-Without the copy the exec dies with `can't open file '/mtp_measure.py': [Errno 2]
-No such file or directory` — measured 2026-10-01, which silently cost a whole
-arm's numbers. A window script must do the copy, not assume the path exists.
-
-### The then-deployed shape, measured again (2026-10-01 A/B window)
-A clean production-environment boot of the config deployed at that time (4 x 262144 fp8, MNBT
-2048 in that window) reported **`GPU KV cache size: 1,079,582 tokens, Maximum
-concurrency for 262,144 tokens per request: 4.12x`** — the same figures as the
-deployed MNBT-1024 shape, i.e. MNBT does not move the pool, and the capacity is
-reproducible across boots.
-
-## See Also
-
-- [[dankserver-frontier-moe-feasibility]] — the sibling feasibility page
-  (which frontier MoE fits this box; the same PCIe-expert-stream wall at
-  V4.1 scale)
-- [[frank-archon-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (the R9V
-  fork era; production has been the tcclaviger fork since)
-- [[x4-pair-experiment]] — the same residency law measured on a narrower
-  link: streaming over PCIe loses, computing in RAM does not care
-- [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here

serverAppConfigs-docs/LLM/dankserver-vram-growth.md

@@ -116,7 +116,7 @@ Verified 2026-08-14 after deploy: flash attention enabled, warmup clean, decode
 
 ## Mitigation Options
 
-The documentation for the benchmark ([[frank-zealot-benchmark]])
+The documentation for the benchmark ([[frank-zealot]])
 prescribes the first option for this crash class. None of these change
 the model.
 
@@ -135,6 +135,6 @@ PR #25649 (slot checkpoints freed on prompt clear, merged), issue #26119
 
 ## See Also
 
-- [[frank-zealot-benchmark]]
+- [[frank-zealot]]
 - [[Dankserver System Specs]]
 - [[deployment]]

serverAppConfigs-docs/LLM/dflash2-large-vulkan.md

@@ -1,174 +0,0 @@
----
-type: reference
-title: DFlash2 on Dankclaw (frank-dragoon)
-description: ARCHIVED - unsloth Qwen3.8-27B-UD-Q8_K_XL served by upstream llama.cpp PR #27342 with a z-lab DFlash2 drafter on the Vulkan backend. Retired 2026-08-21 for speed; kept as a runbook.
-tags:
-  - llama.cpp
-  - vulkan
-  - dankclaw
-  - gfx1151
-  - dflash
-  - dflash2
-  - speculative-decoding
-timestamp: 2026-08-21
----
-
-# DFlash2 on Dankclaw (frank-dragoon) — ARCHIVED
-
-> **Archived 2026-08-21.** This DFlash2 Vulkan deployment was retired the
-> same day it shipped because it was slower than the CIRU runtime it
-> replaced. `frank-dragoon` is back on CIRU. This page stays as the
-> runbook and provenance for the DFlash2 attempt.
-
-`frank-dragoon` (port 52625) briefly served
-`unsloth/Qwen3.8-27B-UD-Q8_K_XL` with a DFlash2 drafter from z-lab. The
-runtime was upstream llama.cpp PR #27342 built with `GGML_VULKAN=ON`.
-
-## Why Vulkan
-
-DFlash2 selects its draft candidates with a lightweight path selector
-built on the `TOP_K` and `ARGMAX` graph ops. The HIP backend does not
-implement `TOP_K`. That is why the retired CIRU setup routed the MTP
-draft around it with `LLAMA_MTP_CPU_ARGMAX_FASTPATH`.
-
-The PR reference fork (Nathanw1014/strix-halo-llamacpp) validated DFlash2
-on this exact gfx1151 through Vulkan (RADV). Its `test-backend-ops` passed
-the new graph ops:
-
-| Op | Cases |
-|----|-------|
-| TOP_K | 445 |
-| CONCAT | 193 |
-| PAD | 28 |
-| REPEAT | 18 |
-| FILL | 4 |
-
-The whole stack (target + drafter) runs on `Vulkan0`.
-
-## What Runs
-
-| Artifact | Repo / File | Size |
-|----------|-------------|------|
-| Target | `unsloth/Qwen3.8-27B-GGUF` `Qwen3.8-27B-UD-Q8_K_XL.gguf` | 31.4 GB |
-| Drafter | `z-lab/Qwen3.8-27B-DFlash2-GGUF` `Qwen3.8-27B-DFlash2-Q4_K_M.gguf` | 2.0 GB |
-| Runtime | upstream llama.cpp PR #27342, GGML_VULKAN=ON | - |
-
-The target was already on disk. The drafter was added for this deploy.
-Both live in `/home/dank/models/`.
-
-## Drafter Hyperparameters
-
-Read from the drafter GGUF metadata:
-
-| Parameter | Value |
-|-----------|-------|
-| `dflash.block_count` | 5 |
-| `dflash.block_size` | 8 |
-| `dflash.conv_kernel_size` | 2 |
-| `dflash.conv_group_size` | 16 |
-| `dflash.selector_rank` | 256 |
-| `dflash.selector_top_k` | 16 |
-| `dflash.context_length` | 262144 |
-| `dflash.embedding_length` | 5120 |
-| architecture | `dflash` |
-
-The target and drafter share the Qwen3.8 architecture
-(`qwen35`, dense 27B, vocab 248320), so the drafter inherits the target
-tokenizer and LM head.
-
-## Build Runbook (2026-08-21)
-
-Performed on Dankclaw.
-
-1. In `/home/dank/llama.cpp`, register the `ggml` remote and fetch the PR
-   head:
-   ```
-   git -C /home/dank/llama.cpp remote add ggml https://github.com/ggml-org/llama.cpp.git
-   git -C /home/dank/llama.cpp fetch ggml pull/27342/head:pr-27342
-   ```
-2. Add a detached worktree that does not disturb the running build:
-   ```
-   git -C /home/dank/llama.cpp worktree add --detach /home/dank/llama.cpp-dflash2 pr-27342
-   ```
-3. Configure with Vulkan only (glslang and `glslc` must be installed):
-   ```
-   cmake -B /home/dank/llama.cpp-dflash2/build-dflash2 -G Ninja \
-     -DGGML_VULKAN=ON -DGGML_HIP=OFF -DGGML_CUDA=OFF -DGGML_METAL=OFF \
-     -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release \
-     -DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_WEBUI=OFF
-   ```
-4. Build the server:
-   ```
-   cmake --build /home/dank/llama.cpp-dflash2/build-dflash2 --target llama-server -j16
-   ```
-5. Verify the Vulkan device:
-   ```
-   /home/dank/llama.cpp-dflash2/build-dflash2/bin/llama-server --list-devices
-   ```
-   Output: `Vulkan0: AMD Radeon Graphics (RADV STRIX_HALO)`.
-
-```mermaid
-graph TD
-    A[Fetch PR #27342 head] --> B[Worktree on DFlash2 head]
-    B --> C[Configure Vulkan build]
-    C --> D[Build llama-server]
-    D --> E[Verify Vulkan0 device]
-    E --> F[Add DFlash2 drafter]
-    F --> G[Swap frank-dragoon]
-    G --> H[Health check + completion]
-```
-
-## Service
-
-- Unit: `frank-dragoon.service`, port 52625
-- Script: `scripts/serve-frank-dragoon.sh`, env `scripts/frank-dragoon.env`
-- Binary: `/home/dank/llama.cpp-dflash2/build-dflash2/bin/llama-server`
-
-### Launch profile
-
-- 262144 context (native max), 1 slot, f16 KV (~16 GiB), 16 context
-  checkpoints, 8 GiB prompt cache
-- DFlash2 draft, block size 8, device Vulkan0, drafter fully offloaded
-- `--spec-type draft-dflash --spec-draft-n-max 8`
-  (clamped to block size 8 - 1 = 7 at runtime)
-- `--temp 0.7 --top-p 0.95`, reasoning on (`--reasoning-format deepseek`)
-- `--no-mmap`, flash attention `-fa on`, `--metrics`
-
-### Required environment
-
-No PromptForge variables are needed. The `TEMP`, `HOST`, and
-`GPU_MAX_HEAP_SIZE` naming rule still applies for portability, even though
-this build uses Vulkan, not HIP.
-
-## Validation
-
-The smoke test on 2026-08-21 used a scratch port (52626) and confirmed:
-
-- Both models load and the server reports healthy.
-- A chat completion returns reasoning and content.
-- The DFlash2 drafter engaged:
-  `draft-dflash: #gen drafts = 11, #acc drafts = 10, #mean acc len = 5.27`.
-- The per-position acceptance curve decays along the block
-  `(0.909, 0.818, 0.727, 0.545, 0.545, 0.364, 0.364)`, as expected.
-
-The published mean acceptance for the Q4_K_M drafter is ~5.39, so 5.27
-is close.
-
-## Operational Notes
-
-- Health: `curl http://127.0.0.1:52625/health`. Returns 503 while loading.
-- Model load takes about a minute with a warm OS page cache (31 GB target)
-  and longer on a cold cache. Do not kill the process during warmup.
-- `deploy.sh dankclaw` waits 20 minutes for health.
-- The Vulkan device reports 113942 MiB. rocm-smi reports the 64 GiB
-  unified-memory budget. Trust the runtime for the addressable SVM pool.
-- The MTP head of the target GGUF is ignored (`--spec-type draft-dflash`
-  replaces it). The load log warns about unused MTP tensors; this is normal.
-
-## See Also
-
-- [[ciru-activefpx-promptforge]] — the active CIRU runtime on Dankclaw
-- [[DFlash DSpark Speculative Decoding]] — DFlash/DSpark design and history
-- [[deployment]] — the deployment and update process
-- [[qwen3.8-strix-optimizations]] — llama.cpp Optimizations on this iGPU
-- [[index]] — knowledge base index

serverAppConfigs-docs/LLM/eemin-rocmi4-mtp.md

@@ -1,188 +0,0 @@
----
-type: reference
-title: caf-eemin Qwen3.8-27B ROCmI4 embedded MTP on Dankclaw
-description: caf-eemin/Qwen3.8-27B-ROCmI4-MTP-GGUF (Q4_0_ROCMI4, embedded NextN/MTP head) served by the charlie12345/ROCmFPX llama.cpp runtime (ROCmI4 W4A4) on the Strix Halo iGPU. frank-dragoon.
-tags:
-  - llama.cpp
-  - rocm
-  - dankclaw
-  - gfx1151
-  - rocmi4
-  - mtp
-  - iu4
-timestamp: 2026-08-25
----
-
-# caf-eemin Qwen3.8-27B ROCmI4 embedded MTP on Dankclaw
-
-> **RETIRED 2026-08-25.** `frank-dragoon` served this model from 2026-08-18
-> until 2026-08-25. It replaced the kingjones777 ROCmFP4 STRIX MTP payload
-> on the same port and the same Strix Halo iGPU, running the new
-> charlie12345/ROCmFPX ROCmI4 W4A4 build (see [[kingjones-strix-mtp]] for
-> the page it replaced), and was itself replaced by
-> [[kingjones-strix-rocmfp4]].
-
-`frank-dragoon` (port 7770) serves
-`cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF` (published under the public
-`eemin` namespace) from the **charlie12345/ROCmFPX** llama.cpp fork built with
-the opt-in `ROCmI4 W4A4` path. The model is `Q4_0_ROCMI4`: signed four-bit
-weights packed two per byte, quantized from the BF16 Qwen3.8-27B and carrying
-its **embedded NextN/MTP head** in the main GGUF — so there is no separate
-draft file. The native gfx1151 `v_wmma_i32_16x16x16_iu4` matrix instruction
-batch-verifies the MTP proposals for higher decode throughput.
-
-## What This Is
-
-**ROCmI4** is the GGUF weight format (signed 4-bit codes). **IU4** is the AMD
-WMMA instruction spelling that consumes them. **W4A4** means four-bit weights
-and four-bit activations during the accelerated matrix multiply. Ordinary
-one-token decode stays memory-bandwidth-bound (MMVQ); batched prompt processing
-and batched MTP target verification use the IU4 MMQ path. W4A4 is a lossy
-activation-quant speed path — treat it as a speed/quality tradeoff.
-
-The model is a conversion/quantization of `Qwen/Qwen3.8-27B` (Unsloth lineage,
-Apache-2.0), not a retrained model. It reports architecture `qwen35`
-(M-RoPE), ~27.32B params, native 262144 context, 866 tensors, ftype 108.
-
-## Model Files
-
-| File | Size | SHA-256 |
-|------|------|---------|
-| `Qwen3.8-27B-Q4_0_ROCMI4.gguf` | 13.50 GiB (14,534,384,384 B) | `ad51ecbee55930ababc629c377943a7bbe50f8aafcf32340c7cdae6fa6b7bb3c` |
-| `mmproj-BF16.gguf` (vision, optional) | 0.87 GiB (931,146,432 B) | `83ee4f4f205fa514161778c41df1ea14144faa0f713510893b63c2395f5c2d53` |
-
-Store under `/home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP/`. The vision
-projector is optional for text-only use.
-
-## Runtime Build
-
-The stock llama.cpp build does not understand `Q4_0_ROCMI4` or the qualified
-IU4/MTP path. Use charlie12345/ROCmFPX at the pinned commit, built HIP-only
-(so the combined HIP/Vulkan scheduler cannot route the ROCmI4 tensors to
-Vulkan, which has no shader for this type) with the W4A4 option on.
-
-| Artifact | Path |
-|----------|------|
-| Fork + pinned commit | `/home/dank/rocmi4-qwen38/ROCmFPX` @ `c49ebdbd5c9f01ec242369f9e7f7967855f80cba` |
-| Build | `cmake -S . -B build-rocmi4-w4a4 -G Ninja -DGGML_HIP=ON -DGGML_VULKAN=OFF -DCMAKE_HIP_ARCHITECTURES=gfx1151 -DGGML_HIP_ROCMI4_W4A4=ON -DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_WEBUI=OFF` |
-| Binary | `/home/dank/rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server` (version 257, `c49ebdb`) |
-| ROCm | system `/opt/rocm-7.2.4`, amdclang 22 |
-
-Build a separate directory so the exact-int8 path (the default branch) stays
-available. Rollback: disable `-DGGML_HIP_ROCMI4_W4A4=ON` and rebuild — no model
-conversion or data change needed. The build prints
-`ROCmI4 W4A4: enabled for device 0` when the accelerated path is active.
-
-## Model Download
-
-Downloaded from the public
-`eemin/Qwen3.8-27B-ROCmI4-MTP-GGUF` repo (the card's download section still
-names the still-private `cafonz/…`; use `eemin`). Verify against the repo's
-`checksums.sha256` after download:
-
-```
-cd /home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP
-sha256sum -c checksums.sha256
-```
-
-## Launch Profile
-
-The qualified MTP profile (from the model card and PR #94, verified on the
-live service): strict Qwen MTP, backend sampling, `n_max 4`, `p_min 0.60`.
-
-```mermaid
-graph TD
-    A[llama-server charlie12345 ROCmFPX c49ebdb<br/>ROCmI4 W4A4 enabled] --> B[load Q4_0_ROCMI4 GGUF + mmproj<br/>-ngl 999 -dev ROCm0]
-    B --> C[draft-mtp on ROCm0<br/>--spec-draft-ngl all]
-    C --> D[strict Qwen MTP<br/>n_max 4 / p_min 0.60<br/>backend sampling]
-    D --> E[Sharp chat template<br/>reasoning deepseek]
-```
-
-Key flags (see `scripts/serve-frank-dragoon.sh`):
-
-```
---spec-type draft-mtp --spec-mtp-strict-qwen
---spec-draft-device ROCm0 --spec-draft-ngl all
---spec-draft-type-k f16 --spec-draft-type-v f16
---spec-draft-n-max 4 --spec-draft-n-min 0
---spec-draft-p-min 0.60 --spec-draft-backend-sampling
-```
-
-Because the MTP head is **embedded**, no `--model-draft` is passed and the
-draft uses all layers on ROCm0. `--spec-mtp-strict-qwen` preserves the target
-path's greedy decisions; it does not remove the W4A4 activation approximation.
-
-### Serving settings
-
-- 65536 context (192K below the native 262144 maximum), 1 slot, f16 KV,
-  context checkpoints, prompt cache. The lower limit leaves headroom for the
-  model plus KV cache in the host's unified-memory budget. Concurrent requests
-  queue; the server never preempts a running generation.
-- `-t 16 -tb 32`, batch 512 / ubatch 256 (the qualified W4A4 profile).
-- `--temp 0.7 --top-p 0.95`, reasoning on (`deepseek` puts `<think>` into
-  `message.reasoning_content`).
-- Qwen-Sharp chat template (`--chat-template-file`), `enable_thinking` +
-  `preserve_thinking` on, preserving the peg-native + deepseek shape the Pi
-  agent expects.
-- Strix Halo unified memory + native gfx1151 detection: `HSA_OVERRIDE_GFX_VERSION=11.5.1`,
-  `GGML_HIP_ENABLE_UNIFIED_MEMORY=1`.
-
-### Service
-
-- Unit: `frank-dragoon.service`, port 7770
-- Script: `scripts/serve-frank-dragoon.sh`, env `scripts/frank-dragoon.env`
-
-## Tuning MTP
-
-MTP gains are content-dependent: predictable code, structured data, and lists
-accept longer drafts than creative prose. The active conservative profile is
-`n4 / p0.60`; the previously qualified starting point was `n16 / p0.60`.
-
-## ubatch Benchmark
-
-Decode throughput depends on the ubatch (per-eval token batch). Run the sweep
-on 8k contexts, 512 → 4096 in steps of 512, with a fixed batch:
-
-```
-sudo systemctl stop frank-dragoon
-./scripts/bench-ubatch.sh
-sudo systemctl start frank-dragoon
-```
-
-The script stops `frank-dragoon` for a clean GPU, boots one server
-instance per ubatch on a scratch port, reads authoritative `print_timing` tok/s,
-and restarts the service. See [[frank-dragoon-benchmark]].
-
-## Known Issues
-
-1. **gfx1151 only** for the W4A4 path. Other GPUs fall back to exact int8 MMQ.
-2. **Memory pressure:** the model is 13.5 GiB. Under `--no-mmap` with full
-   offload it needs the whole tensor set at once; leave room in the 64 GiB
-   unified pool (rocm-smi figure). Do not run a bench with the live service up
-   or it OOMs at load.
-3. **`Q4_0_ROCMI4` + MTP flags need the ROCmFPX fork**; stock llama.cpp may
-   reject the file or drop the optimized path.
-4. **W4A4 is lossy** (+~5.4% perplexity vs exact int8). Opt-in speed path.
-
-## Measured Results (qualification)
-
-From the model card, gfx1151, pinned build, W4A4 + strict MTP-16:
-
-| Result | Value |
-|--------|-------|
-| HumanEval pilot, W4A4 IU4 (mean) | 49.40 tok/s |
-| Full HumanEval, W4A4 (mean / median) | 44.39 / 45.23 tok/s |
-| HumanEval / HumanEval+ pass@1 | 94.5% / 91.5% |
-| Non-speculative tg128 | ~13.8 tok/s |
-
-Local smoke test 2026-08-25 on the iGPU: model loaded, strict-MTP active,
-draft acceptance 0.74 (mean 7.67), generation ~25 tok/s at a small output.
-The ubatch sweep above gives the production ubatch choice.
-
-## See Also
-
-- [[deployment]] — deploy and update process
-- [[kingjones-strix-mtp]] — the retired ROCmFP4 predecessor on this slot
-- [[ciru-activefpx-promptforge]] — the retired v3 CIRU PromptForge page
-- [[Dankclaw System Specs]] — the box
-- [[frank-dragoon-benchmark]] — backend/spec benchmark history

serverAppConfigs-docs/LLM/frank-archon-freetoken-pr587.md

@@ -1,184 +0,0 @@
-# frank-archon — FreeToken PR #587 ROCm (Qwen3.8-Flash-Next-Uncensored-FP8, 1× R9700)
-
-**Status: RETIRED 2026-10-05 → (superseded the same day by the radiance
-cutover — see [[frank-archon-radiance-qwen38-native-fp8]], itself retired
-2026-10-08; the payload live today (2026-10-18) is the publisher iq4r container,
-[[frank-archon-radiance-qwen38-iq4r]]). Kept as the rollback runbook: the image
-is still in dank's rootless storage, the orcarouter payload is not on disk any
-more)** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 ·
-**GPU:** 1× Radeon AI PRO R9700 (gfx1201) — the second R9700 sits idle
-
-The tcclaviger RDNA4 vLLM fork was replaced by **FreeToken at PR #587 head
-`77a7201f98f00d93f618c6cf8b81ffe8360e0899`** (branch `rocm714_torch214`), built
-on the host into the local image `localhost/frank-primary-freetoken:pr587-77a7201`
-(that is the tag podman reports today — the 2026-10-10 rename never retagged it)
-by `scripts/docker/frank-primary-freetoken.Dockerfile`, which survives only in
-the rollback tag: `scripts/docker/` is not in the tree today. Same port, same
-served alias (the wire id was `frank-primary` until the 2026-10-09 rename), same
-131-shard payload (`/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8`, since
-deleted from the host). Rollback:
-git tag `frank-archon-vllm-shapeG-20261003` (the vLLM trio + its env lore),
-runbook [[dankserver-qwen38-fp8-expert-offload-tuning]].
-
-## Why FreeToken
-
-The vLLM fork planned a per-boot VRAM/host tiering from an estimator: util
-0.982 was a hard law (1.00 was **unbootable** — worker init raises
-`ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` when budget ==
-total), the expert arena capped at 17.77 GiB / 7,762 slots, and live traffic
-still degraded ~40k tokens in (the GDN state-fault class, never fixed in the
-fork). FreeToken keeps **all expert weights in pinned host RAM** with an LRU
-GPU cache, sizes **KV as an explicit token pool**, and boots in one
-deterministic pass — no per-boot estimator to fight.
-
-## The build (on dankserver, rootless podman as dank)
-
-```bash
-git checkout frank-archon-freetoken-pr587-20261005   # scripts/docker/ exists only in the tag
-cd /home/dank/serverAppConfigs
-podman build -t frank-primary-freetoken:pr587-77a7201 \
-  -f scripts/docker/frank-primary-freetoken.Dockerfile scripts/docker/
-```
-
-(The names above are the ones the tag actually carries —
-`scripts/docker/frank-primary-freetoken.Dockerfile`, per
-`git ls-tree frank-archon-freetoken-pr587-20261005`. The 2026-10-10 rename
-swept the docs, not the tag or the image in storage.)
-
-Laws baked into the Dockerfile (each one cost a failure):
-
-- **Base digest-pinned**: `rocm/pytorch:rocm7.14_ubuntu24.04_py3.12_pytorch_release_2.11.0@sha256:a223aee1…` —
-  the PR doc's known-good stack (torch 2.11.0+rocm7.14, triton 3.7). No
-  rocm7.14+torch2.14 image exists on Docker Hub (the 2.14 images are
-  rocm10.1.0 — an untested pair for this PR).
-- **`--no-build-isolation --no-deps -e .`** — a plain
-  `pip install -e .` on ROCm can REPLACE the image's ROCm torch with a CUDA
-  build (the dependency set is authored for NVIDIA). Runtime deps are
-  curated explicitly; `flashlib==0.3.0 --no-deps` (its numba /
-  nvidia-cutlass-dsl deps are CUDA-only); ninja for the QSA JIT;
-  **flashinfer / sgl_kernel intentionally absent** (no ROCm builds;
-  FreeToken falls back to its Triton kernels — the PR's 12 pytest failures
-  are exactly those backends).
-- **`libhiprtc.so` dev symlink**: the pip-ROCm torch ships only the
-  versioned `libhiprtc.so.7` in
-  `/opt/venv/.../site-packages/_rocm_sdk_core/lib`, but the PR's `_cpu_moe`
-  extension links `-lhiprtc`. First build died `ld: cannot find -lhiprtc`;
-  the Dockerfile symlinks the unversioned name into the ALREADY-LINKED dir.
-- FreeToken source is checked out at the **pinned PR head**, not the branch
-  (a future push to `rocm714_torch214` cannot change the deployed engine).
-- The stale Docker Hub login (`docker login` creds at
-  `~/.docker/config.json`) poisons podman's ANONYMOUS pulls of public images
-  with "invalid username/password" — `docker logout` fixes the pull. Hit
-  2026-10-05 while pulling the base image.
-
-## Measured at cutover (2026-10-05/06)
-
-
-- **Boot**: ~2.3 min container-start → API-ready (131 shards page-in + PLE
-  pin + expert staging + CUDA graphs `[1,2]`). VRAM after graphs:
-  **1.80 GiB free** of 31.86. RAM: 122 GB used + **113 GB shared (the
-  pinned expert/PLE slabs)**, 123 GB still available.
-- **Decode: 17.1 tok/s** single-stream, temp 0, 400-token probe (clean
-  1→N counting). The vLLM incumbent measured ~30 tok/s on TWO cards;
-  FreeToken single-card is the deployed baseline.
-- **The single-card LRU ceiling**: the KV contract (harness worst case
-  177,356 tokens) + dense fp8 + the vision tower + graphs leave ~12 GiB for
-  the expert LRU at rate 0.10 — and only 1.8 GiB of live headroom, so
-  `ft ctl cache --moe` cannot raise it far on one card. The paths to a
-  better decode rate are TP=2 (below) or a smaller pool (contract
-  violation — the 177,356 floor is real traffic, not padding).
-- **TP=2 exists but does not boot on ROCm** (probed 2026-10-06,
-  zero-impact window): `--tensor-parallel-size 2 --gpu 0,1` dies in the TP
-  allreduce JIT — `pynccl.cu` compiles per-rank at boot and clang can't
-  resolve `cuda_runtime.h` from the hipified include path
-  (`1 error generated when compiling for gfx1201`, ninja stopped). The PR
-  author validated single-GPU only; this is FreeToken's untested-TP-on-ROCm
-  edge, not a board or driver fault (radiance TP=2 allreduces fine on this
-  pair since the 2026-09-20 swap). A future fix = patch the JIT include
-  path (HIP's cuda_runtime shim) or vendor an ROCm allreduce — re-probe
-  then.
-- **Verification gates passed**: /health ok; plain completion `FLEET-OK`
-  (finish stop); thinking split — `reasoning_content` non-empty AND
-  `content` carries the answer (17×24 → "408"); native `tool_calls` with
-  correct name+arguments (finish tool_calls); 256×256 solid-red PNG through
-  an `image_url` part answers exactly "Red" (prompt_tokens 127); stats:
-  native ctx 262,144, KV 4,687 pages × 64 = 299,968, modalities
-  [text,image], sampling defaults from generation_config.json
-  (temp 1.0 / top_p 0.95 / top_k 20).
-- **Thinking counts against max_tokens** (same as the vLLM era): a
-  `max_tokens: 10` probe returns EMPTY content with the budget spent in the
-  think block. Clients must budget for thinking or disable it.
-- **Prometheus scrapes /metrics → 404** (Dankclaw scrapes 192.168.0.235:7770
-  from the llm-fleet job; FreeToken has no /metrics endpoint). The target
-  will read down until the scrape config is repointed or a metrics shim
-  lands — same class as the retired HTML-probe targets
-  (see [[Observability]]).
-
-
-## The shape
-
-| Knob | Value | Why |
-|---|---|---|
-| `--gpu` / `HIP_VISIBLE_DEVICES` | 0 / 0 | ONE R9700; visibility pinned so in-container index 0 is the right card |
-| `--num-tokens` | **299,968** | KV pool in TOKENS; **must be a multiple of 64** (the auto attention backend is `qsa_sparse`, which overrides page size to 64 — 300,000 was refused: `nearest valid values: 299968 or 300032`) |
-| `--max-running-requests` | 2 | first-boot conservative; raise after real-traffic slot-occupancy look |
-| `--max-output-tokens` | 36,864 | harness maxTokens; requests that omit max_tokens get this default |
-| `--moe-cache-rate` | 0.10 | GPU expert LRU as fraction of all experts (~120 GiB fp8 → ~12 GiB VRAM); first-boot conservative |
-| parsers | AUTO | thinking → `reasoning_content` natively; tools in the qwen3.8 dialect; NO `--reasoning-parser`/`--tool-call-parser` flags |
-
-KV math: 12 full-attention layers × 2 KV heads × 256 head_dim × 2 (K+V) ×
-2 B = **24 KiB/token** (f16 KV; the 36 GDN layers use fixed per-seq state,
-not per-token KV) → 299,968 × 24 KiB ≈ 7.0 GiB. Clears the harness worst
-case (140,492-token prompt + 36,864 max_tokens = 177,356) and admits ONE
-native 262,144-token conversation + full budget. `models.yml`
-`contextWindow: 200704` stays (compacts early, never 400s).
-
-## RAM math (why the root unit + memlock)
-
-246 GiB total. FreeToken pins ~120 GiB of fp8 expert rows + the ~47.7 GiB
-PLE ngram table in host RAM → the unit carries
-`LimitMEMLOCK=infinity` (root unit; dank's login session caps at 8 MiB and
-a user quadlet cannot raise it). Weights+PLE+overhead ≈ 190 GiB — the vLLM
-container's ~170 GiB MUST be down first; `systemctl stop frank-archon`
-frees it (measured: 207 GB free right after the stop).
-
-## Resize live
-
-```bash
-ft ctl cache --kv <tokens>   # grows the KV pool LIVE (shrink needs restart)
-ft ctl cache --moe <rate>    # same for the expert LRU
-ft ctl stats                 # pool occupancy, VRAM, throughput
-```
-
-## Verification (the fleet's real-completion law)
-
-1. `/health` 200 — necessary, not sufficient.
-2. Plain completion returns the asked-for string.
-3. Thinking request: `reasoning_content` non-empty AND `content` carries the
-   answer (the models.yml `reasoningContentField: reasoning_content` flip
-   rides on this).
-4. Vision probe: a solid-color PNG through an `image_url` part answers the
-   exact color (the legacy top-level `images` field trap does not exist
-   here — FreeToken is OpenAI-native).
-5. Unit-managed: `systemctl show -p MainPID` behind the health answer (the
-   2026-09-26 stray-port false green).
-
-## Rollback
-
-```bash
-git checkout frank-archon-vllm-shapeG-20261003 -- \
-  scripts/frank-archon.env scripts/serve-frank-archon.sh \
-  scripts/frank-archon.service
-./tests/run.sh && git commit -m "frank-archon: rollback to vLLM shape G" \
-  && git push && ./scripts/deploy.sh dankserver --service frank-archon
-```
-
-The tcclaviger image and the FreeToken image coexist in dank's rootless
-storage; the deploy re-pulls the pinned digest if absent.
-
-## Index
-
-- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the retired vLLM shape
-  (rollback target; its 362-line env lore lives in the tag)
-- [[freetoken-frank-small]] — the v0.1.2 CUDA precedent (ortobox, RTX 3090)
-- [[LLM/index]] — fleet index

serverAppConfigs-docs/LLM/frank-archon-glm53-flash-w4a16.md

@@ -1,808 +0,0 @@
----
-type: runbook
-title: frank-archon — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
-description: VERIFIED 2026-09-30 by boot probe, bench, live expert-cache telemetry AND a torch-profiler kernel-time window. frank-archon cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET, and the profiler window FALSIFIED the first explanation: the expert gather is 1.05 % of device time, not the bound. 44 % of the 93 ms/token budget is pointwise glue (1,428 elementwise launches per token) because the fork's GLM trunk is NOT torch.compiled while its own Qwen trunk is; the second-largest lever, the fused router, is binary-blocked at 288 experts. The compile changelist is staged but incomplete (three patches, one blocker left).
-tags:
-  - dankserver
-  - r9700
-  - gfx1201
-  - glm5.3
-  - w4a16
-  - int4
-  - expert-offload
-  - expert-cache
-  - autoround
-  - vllm
-timestamp: 2026-09-30
----
-
-# frank-archon — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
-
-> **RETIRED 2026-10-01** (era 2026-09-30 → 10-01, matching the AGENTS.md Runbook
-> Index). Cutover reversed to Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 — the
-> trio as of commit `2339fae`, restored by `d91379c` (2026-09-30). frank-archon
-> runs [[frank-archon-radiance-qwen38-iq4r]] (radiance) today, 2026-10-18. The
-> DFlash2 drafter experiment is a **failure**: after five patches the drafter
-> served and drafted (12.8 tok/s proposed, 65,536 ctx, both TP ranks capturing
-> FULL dflash2 cuda graphs) but **acceptance pinned at 1.00-1.38 on every text
-> class including verbatim repetition** — the signature of a mis-wired
-> aux-hidden-state capture, not a weak drafter. At ~1.2 acceptance the 8-token
-> verification tax makes decode 5x SLOWER than the 9.9-10.0 no-draft baseline
-> (1.6-1.8 tok/s measured). The untested alternative capture semantics: post-layer
-> output state (DeepSeek's `hidden_states + residual`) or the mHC `comb` stream.
-> Payload (170 GiB) + drafter (2.2 GiB) were staged to
-> `/home/dank/.AGENTTRASH/20261001-glm-purge/` and purged since — on 2026-10-18
-> Dankserver has no GLM tree under `/home/dank/models/` and `~/.AGENTTRASH` is
-> empty. KV-cache work that survives: `glm53-dflash-04-kv-groups-draft.patch` and
-> `glm53-dflash-04b-kv-registry-dflash.patch` are still in `scripts/patches/`;
-> patch 03 (tailpin) was deleted at the rollback (`d91379c`) and patch 02 was
-> never committed. None of these apply byte-exact today — the 2026-10-10 rename
-> (`55f87fc`) rewrote the retired service names inside their context lines.
-
-
-> **Question this page answers.** Can the fleet serve the 320B-class
-> `GLM-5.3-Flash` at int4 on two R9700s with expert offload, and how fast is it?
-> **Answer.** Yes — it serves, with vision, tool calls and 131,072 ctx. It runs
-> at **9.9-10.0 tok/s decode / 918 tok/s prefill**, and **35 tok/s is not
-> reachable** on this payload as the fork ships it. The bound is **not** the
-> model, the quant, the link, or the expert gather (measured: 1.05 % of device
-> time). It is the **unfused pointwise glue of an uncompiled trunk** — 44 % of a
-> 93 ms per-token budget spread over ~1,428 elementwise launches per token —
-> because the fork compiles its own Qwen trunk and not its GLM one. See
-> [The measured profile](#the-measured-profile-and-the-real-wall).
-
-## DFlash2 target-side port — the four laws and the KV ladder
-
-Distilled 2026-10-05 from the probe windows (skills `glm53-dflash-boot-ladder`
-/ `glm53-dflash-eagle3-port`); each law was proven by a real failed boot.
-
-**The four EAGLE3-port laws** (make the GLM trunk satisfy `SupportsEagle3`):
-
-1. **Patch the instance the runner calls.** `model_runner.py:425` calls
-   `set_eagle3_aux_hidden_state_layers(self.model)` on the *loaded* model —
-   GLM loads as `Glm5NextForConditionalGeneration` (multimodal wrapper;
-   `self.language_model = Glm5NextForCausalLM`). Members placed only on the
-   CausalLM fail the check on the conditional — same error text, wrong file.
-   Patch BOTH wrappers.
-2. **Presence-checked member set.** `SupportsEagle3` (interfaces.py:1659) is
-   `@runtime_checkable`; `isinstance` walks ALL `__protocol_attrs__`
-   (including base-protocol members) with `getattr_static`, breaking on the
-   first miss. Both wrappers need, as class attributes:
-   `supports_eagle3: ClassVar[Literal[True]] = True`,
-   `has_own_lm_head: bool = False`, `has_own_embed_tokens: bool = False`.
-   Verify in-image with stock typing — no boot needed.
-3. **MRO forbids inheriting the Protocol.** GLM wrappers carry four/five
-   Protocol-derived bases; adding `SupportsEagle3` anywhere dies with
-   "Cannot create a consistent method resolution order" (sim-proven: the
-   ordering rejects it in every base position). The attribute route avoids
-   the MRO, so the two methods must be written by hand: the CausalLM
-   unwraps to `self.model` (the `EagleModelMixin` trunk), the conditional
-   delegates to `self.language_model` (pixtral.py:498 pattern);
-   `get_eagle3_default_aux_hidden_state_layers` returns `(2, L//2, L-3)`,
-   with the drafter config's `target_layer_ids` winning when present.
-4. **Trunk capture point:** append `aux_hidden_states` BEFORE the layer call
-   — GLM's loop carries the single-stream state entering the layer.
-
-**The KV blocker ladder** (patches `scripts/patches/glm53-dflash-0*`):
-
-1. EAGLE3 gate → presence-checked attrs on both wrappers (above).
-2. KV page-size unifier: MLA layers cannot be padded (sparse MLA reads
-   whole token rows) but CAN grow block_size when the max page is an exact
-   multiple — bump `max_page_size` in `unify_kv_cache_spec_page_size` to
-   the `lcm` of all MLA-family pages first (patch 02).
-3. kpool tail assert `tail_kv_cache.shape[2] == pool_size`: the unifier's
-   divisible branch grew KpoolTailSpec's block; its kernel pins block ==
-   index_kpool. Add a KpoolTailSpec branch that pads `page_size_padded`
-   instead of growing (patch 03).
-4. KV capacity: the glm5 layout ALIASES mamba states INTO MLA pages (why
-   GLM fits 32 GiB cards); the generic path (taken when draft/hidden specs
-   exist) cannot alias → ~30 GiB demanded vs a 0.92 GiB reserve. Extend
-   `_get_kv_cache_groups_glm5_next` to partition
-   HiddenStateCacheSpec + drafter attention specs; the drafter forms its
-   own uniform group (per-group pages, NO unification → the MLA block
-   stays at the engine-auto-set 2,432 from interface.py:918, kpool guard
-   `block_size % 512 == 0` unstressed); layout returns a 10-tuple
-   (+draft_group, +hidden_groups); the builder sizes ONE shared backing
-   allocation (`allocate_kv_cache` asserts all KVCacheTensor sizes equal) =
-   aliased pool + appended draft/hidden tensors (patch 04). After 04 the
-   capacity requirement collapsed 31.76 → 1.88 GiB at 65,536 max len and
-   the engine SERVED (READY ~500 s, dflash2 fused prep ON, drafter FULL
-   graphs on both TP ranks).
-5. Reserve registry: `_glm53_kv_bytes_per_token` must add 4,096 (drafter
-   KV) + 5×hidden×2 (aux hidden cache) keyed on
-   `speculative_config.method in {dflash, dspark, extract_hidden_states}`.
-   NEVER key on `use_aux_hidden_state_outputs` — it lives on the
-   gpu_model_runner instance (gpu_model_runner.py:623/678), unset at plan
-   time, so the check silently fails and the reserve stays 13,600
-   (patch 04b). Proven reserve math at 65,536: 13,600 + 4,096 +
-   5×4096×2 ≈ 58,656 B/token → ~4.15 GiB reserve, arena ~10.3 GiB.
-
-**Verify cheaply before booting:** smoke-test the pure KV logic on CPU
-torch with the image's real spec classes — stub `vllm.logger`/`vllm.utils.*`,
-register the spec classes in `KVCacheSpecRegistry` with a dummy manager, add
-`from __future__ import annotations`, and run synthetic GLM+draft+hidden
-spec sets through the patched grouping/layout/builder functions. A 7-minute
-boot per hypothesis is the alternative.
-
-**Window discipline:** stop + mask `frank-archon` FIRST and gate on
-MemAvailable ≥ 230 GiB before booting the probe (the expert-offload planner
-reads MemAvailable; beside-prod boots die at plan time with `ExpertPlanError:
-host holds 5113 of 12096 swappable experts`). Save `podman logs glm-probe`
-to a file BEFORE teardown — a `podman logs | grep | head` pipeline died once
-under pipefail via SIGPIPE before log-save, losing the capture. The abort
-path must restore prod (unmask + daemon-reload + reset-failed + start),
-never bare-exit after stop+mask. Launcher knobs: `DFLASH_PATCH=1` (mounts
-the patched model.py), `DRAFT_MOUNT=1`, `SPECJSON_FILE=…` ({"method":
-"dflash","model":"/draft","num_speculative_tokens":8}) — note the launcher's
-echo prints `patch=$PATCH_MOUNT` (the COMPILE knob), not DFLASH_PATCH.
-Boot timeline: EAGLE3 gate fires ~4 min in at worker init; drafter
-torch.compile is the frontier (~380 s).
-
-**Open telemetry discrepancy:** the contaminated window read 1.5–2.1 tok/s
-with acceptance 1.00–1.25; Frank live-measured ~18 tok/s decode / ~370 tok/s
-prefill. Reconcile with a clean single-request measurement before trusting
-either.
-
-Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the same
-hardware — its conclusion was drawn from throughput arithmetic and is superseded
-by this page's kernel-time measurement),
-[[dankserver-frontier-moe-feasibility]] (the earlier fit analysis),
-[[VLLM Index]], [[LLM/index|LLM fleet]].
-
-## The deployed shape
-
-| term | value |
-| --- | --- |
-| payload | `Intel/GLM-5.3-Flash-W4A16-AutoRound` — AutoRound **symmetric int4, group 128**, packing `auto_round:auto_gptq`, 34 shards + 2 extra safetensors, 113,074 tensors, ~170 GiB |
-| runtime | `docker.io/tcclaviger/vllm@sha256:da4029666…` (same digest-pinned image as the qwen cutover) |
-| engine | TP=2, `--enable-expert-offload --expert-precise-memory`, max-model-len 131,072 x 2 seqs, bf16 KV, util 0.98, NBATCH 2048, prefix caching |
-| MoE kernel | `R4dW4a16MoEMethod` → libr4d grouped W4A16 GEMM, 6.234 MiB per expert-rank slot |
-| expert cache | 14.51 GiB = **2,383 slots** per rank; 12,096 host-backed experts, **73.64 GiB pinned per rank** |
-| unit | `scripts/frank-archon.service` (root unit, `LimitMEMLOCK=infinity` for the pin) |
-| measured | **decode 9.9-10.0 tok/s**, **prefill 918 tok/s** (warm, 2.6k-token prompt), KV cache 138,561 tokens |
-
-```mermaid
-graph LR
-    A[Intel AutoRound W4A16<br/>sym int4 g128, auto_gptq] -->|quant_method auto-round| B[inc.py override to inc]
-    B --> C[inc_wna16_scheme<br/>_resolve_gptq_moe]
-    C -->|sym int4 g128 + gfx1201 + libr4d| D[R4dW4a16MoEMethod<br/>grouped W4A16 GEMM]
-    D --> E[ExpertLRUCache<br/>2,383 VRAM slots / rank]
-    D --> F[73.64 GiB pinned host rows / rank]
-    E --> G[decode step]
-    F -->|host misses<br/>1.05% of device time| G
-```
-
-## Why this payload and not another
-
-The W4A16 fast path on this fork is reachable **only** through the INC
-(AutoRound) quant method. `compressed_tensors` checkpoints of the same model —
-for example `dealignai/GLM-5.3-Flash-CYBERSECURITY-W4A16`, 182 GiB — resolve
-their MoE through the WNA16 oracle to `TritonWNA16Experts`, which **has no
-offload recipe**, and the engine refuses to serve at all: 170 GiB of experts
-cannot fit 64 GiB of VRAM, so offload is mandatory, not optional.
-
-The chain that makes Intel's checkpoint work, with the exact evidence:
-
-```
-[expert_plan.py]  quantization=inc            ← inc.py:364 overrides "auto-round"
-[inc_wna16_scheme.py:249]  INC GPTQ MoE on ROCm: R4dW4a16MoEExperts (libr4d grouped W4A16 GEMM)
-[r4d_w4a16_moe.py:682]     Using MoEPrepareAndFinalizeNoDPEPModular / R4dW4a16MoEExperts
-[expert.py:450]            Expert offload: 42 MoE layers staged; 2,383 cache slots in 14.51 GiB planned per rank
-[expert.py:413]            42 layers split: 0 VRAM-only experts, 2,383 swap slots, arena 14.51 GiB,
-                           host 73.64 GiB pinned, host rows prepacked on the GPU
-```
-
-`R4dW4a16MoEExperts` is what the offload machinery wraps: the offload plan is
-format-agnostic but requires the experts class to expose the hot/cold split.
-
-## The numbers, and the wall
-
-Measured on the deployed engine, 2026-09-30 (`bench_glm.py`, real prose, greedy):
-
-| depth | prefill | decode |
-| --- | --- | --- |
-| 2,621 tok prompt, run 1 (cold cache) | 259 tok/s | 9.88 |
-| 2,621 tok prompt, runs 2-3 (warm) | 918 / 885 tok/s | 9.91 / 9.41 |
-| 493 tok prompt (telemetry window) | 124 tok/s | 8.90 |
-
-Decode is flat across context and across every arena change — the signature of a
-per-step cost that is not KV- or residency-driven at the margin.
-
-### The measured profile, and the real wall
-
-**This section replaces an earlier explanation, which the measurement falsified.**
-The first reading of this page bounded the step with throughput arithmetic: 42 MoE
-layers x 8 experts x 6.234 MiB = 2.045 GiB per step per rank, 0.82 GiB of it
-missed at a measured 0.53-0.66 hit rate, giving "0.82 GiB in ~100 ms = 8.2 GB/s =
-the gather is the wall". That arithmetic was circular — it *assumed* the gather
-consumed the whole step — and it is wrong. The engine, asked directly, disagrees:
-
-A torch-profiler window (`--profiler-config`, `POST /start_profile` /
-`stop_profile`, 496 decode tokens at the deployed shape, rank0 trace parsed for
-device-activity time) gives the true split. Total device kernel time **46.18 s /
-496 tokens = ~93 ms per token** against a measured ~104 ms step (9.6 tok/s with
-the profiler attached):
-
-| ms | % | launches | kernel |
-| --- | --- | --- | --- |
-| 16,074 | **34.8** | 708,500 | `vectorized_elementwise_kernel` (22.7 us each) |
-| 4,602 | 10.0 | 42,000 | `r4d_gemm_moe_w4a16_nt_b16_m64` — the MoE GEMM |
-| 3,108 | 6.7 | 174,708 | `elementwise_kernel_manual_unroll` |
-| 2,943 | 6.4 | 138,252 | `wvSplitK_hf_sml_` — split-K GEMM (projections / head) |
-| 2,829 | 6.1 | 83,160 | router: `sbtopk::gatherTopK` + `warpMergeSortTopK` + `bitonicSortKVInPlace` (1,208 + 1,051 + 559 ms; the 1,051 row is not in the list below) |
-| 3,015 | 6.5 | 133,920 | `r4d_mhc_pre_finish/pre_partial/post` — hyper-connections |
-| 2,782 | 6.0 | 127,916 | inductor `triton_poi_*` / `triton_per_*` (the fusions that DO happen) |
-| **487** | **1.05** | 20,664 | **`r4d_moe_lru_gather_kernel`** — the accused |
-| 250 | 0.54 | 20,832 | `_moe_lru_tele_kernel` — the telemetry that measured it (TELEM was on) |
-
-So the gather is **1.05 %** of device time, and `r4d_moe_lru_fused_kernel` (the
-LRU *manager*, not the copy) is a further 1,599 ms / 3.5 %. Residency work and the
-PCIe link are not the story: 44 % of the budget is **pointwise glue** — bf16<->fp32
-conversions, adds, `pow`/`rsqrt` from fp32 RMSNorm, `launch_clamp_scale` quant
-passes, `reduce_kernel` means — spread over **~1,428 elementwise launches per
-token** (3,734 kernel launches per token in total). The GPU is busy 86 % of the
-wall clock, so this is real device work, not launch-gap overhead.
-
-Two consequences that kill three whole families of planned work:
-
-- **Spec decode cannot help this shape, and the MTP measurement was not an
-  accident.** Pointwise cost scales with the token count, so a draft batch that
-  verifies 4 tokens pays ~4x the glue for the same per-token output — which is
-  exactly why MTP4 measured "acceptance 3.76-4.90, decode unchanged". Same reason
-  the gather was never the bound.
-- **Arena size and hit rate are not levers.** Confirmed twice: `--expert-precise-memory`
-  (arena 13.08 -> 14.51 GiB) moved decode by nothing, and the gather is 1 %.
-
-## Fork work: the compile changelist (WIP, does not boot yet)
-
-The fork compiles its own flagship trunk — `vllm/models/qwen4_exp/amd/model.py`
-carries `@support_torch_compile` — and **not** its GLM trunk. The engine says so on
-every boot:
-
-```
-Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
-provides neither a compiled submodule nor breakable CUDA graphs.
-Overriding to FULL_DECODE_ONLY.
-```
-
-Inductor fusion is therefore the only lever that touches the 44 % pointwise mass.
-Three patches are staged in `scripts/patches/glm53-compile-0*.patch`, each carrying
-its own evidence header, base/patched sha256 and the next blocker:
-
-| patch | target | fixes |
-| --- | --- | --- |
-| `-01-model-decorator` | `models/glm5next/common/model.py` | decorates `Glm5NextModel` (env `R4D_GLM_COMPILE`, default on) |
-| `-02-trace-safe-probes` | `model_executor/kernels/r4d_lib.py` | freezes 10 capability probes that do `os.path.exists`/`.so` lookups **inside the traced forward** |
-| `-03-fullgraph-optout` | `compilation/wrapper.py` | makes `fullgraph=True` opt-out (env `VLLM_COMPILE_FULLGRAPH=0`) |
-
-What the boots established, in order (each one ~7 min):
-
-1. **01 alone dies** on the first cold probe call inside the traced region:
-   `Dynamo does not know how to trace the builtin 'posix.stat'` —
-   `r4d_mhc_supported -> r4d_lib.r4d_mhc_enabled -> available -> os.path.exists`.
-   **`functools.lru_cache` does not protect these probes**: dynamo re-traces the
-   wrapped body to build its guard even when the cache is warm (it warns about
-   exactly this), verified in-image — a *warm* lru_cache-calling function still
-   fails `fullgraph=True`, while a plain module-global read compiles.
-2. **01+02 get past it** (`libr4d capability probes frozen: 10 warm [...]` in all
-   four processes) and die on
-   `RuntimeError: Graph breaks are not supported with aot compile`, because this
-   build runs AOT compile by default and AOT refuses breaks.
-3. **01+02+03 with `VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0` traces the
-   trunk successfully** (`Dynamo bytecode transform time: 1.08 s`) and then dies on
-   the decode forward with `KeyError: 'in_proj_qkvbfg_a'` (`Glm5NextModel.forward`
-   line 748 calling the decoder layer at line 456). `in_proj_qkvbfg_a` is the KDA
-   layer's merged column-parallel projection (`kda.py:217`, called at :407) — a real
-   submodule attribute name, also present in the `load_weights` shard table
-   (`model.py:780-785`). Reading the full traceback (the reported line is inside
-   dynamo-rewritten code) traces the throw to dynamo's own name→module map:
-   `torch/_dynamo/output_graph.py` registers into `nn_modules` at :1259/:1384,
-   looks up with `self.nn_modules[node.target]` at :3000 (a bare-name `KeyError`
-   when absent), and documents stripping `_buffers[..]/_parameters[..]/_modules[..]`
-   from those names at :1225. So the compiler is failing to **resolve a submodule by
-   name** while tracing the GLM decoder layer; whether the root cause is the fork's
-   module tree or dynamo's registration is not yet settled. The next step is the one
-   torch itself suggests in that error — `TORCHDYNAMO_VERBOSE=1` for the internal
-   stack trace (and note the launcher resets `ENVARGS`, so it must be added inside
-   `serve_glm_autoround.sh`'s COMPILE_PATCH branch rather than in the outer
-   environment).
-
-4. **04 removes the fused-norm call from the traced region** (`R4D_GLM_FUSE_QKV_RMSNORM=0`:
-   the GLM MLA block stops calling the fork's fused-norm JIT kernel, whose kernel
-   object in `models/common/ops/fused_qk_rmsnorm.py` is bound *after* the function
-   that uses it — `_FUSED_Q_KV_RMSNORM_KERNEL` at line 223 vs its use at line 211).
-   **It removes a real break** (under `fullgraph=True` the fused call was the first
-   break dynamo reported; after this patch the report moves on to the workspace
-   acquisition) — but it is not by itself what blocks a boot, because with breaks
-   genuinely tolerated the blocker is the name-map `KeyError`. See the run history
-   below.
-5. **A launcher bug that made four boots misleading — worth knowing because the
-   failure kept moving.** The scratch launcher's compile branches were restructured
-   on 2026-09-30 and the `-e VLLM_COMPILE_FULLGRAPH=0` line was dropped, so four
-   consecutive boots mounted the wrapper patch while `_FULLGRAPH` still defaulted to
-   True. With `fullgraph=True` **every** break is an error, so those boots reported
-   whichever break came first:
-
-   | config | outcome |
-   | --- | --- |
-   | `fullgraph=False`, AOT off | breaks tolerated → dies later on the dynamo name-map `KeyError: 'in_proj_qkvbfg_a'` |
-   | `fullgraph=True`, AOT off | dies on the first break (cell → workspace → compiler-disable) |
-
-   So the reported breaks are real *as breaks*, but they are not what blocks a
-   break-tolerant compile — and an earlier version of this page blamed
-   `TORCH_LOGS=graph_breaks` for the difference. **That was wrong**; the flag only
-   logs. The launcher sets the env var again.
-
-**How to see the breaks (and its trap).** `TORCH_LOGS=graph_breaks` names each break
-with a user stack, and `VLLM_USE_AOT_COMPILE=0` is required because AOT compile
-hard-refuses breaks — but a fullgraph=True run reports the first break as fatal, so
-check which configuration you are actually in (`podman exec <ctr> printenv
-VLLM_COMPILE_FULLGRAPH`) before concluding a break blocks the boot. For the name-map
-KeyError there is a ready-made probe: put `/home/dank/dynamo-probe` on `PYTHONPATH`
-with `VLLM_DYNAMO_DUMP_ATTRS=1` (a `sitecustomize.py` — a `.pth` on `PYTHONPATH` is
-NOT executed, and the image's own `sitecustomize` is chained so its apport hook
-survives) and it prints, at the miss, the node target, the near-miss keys, and
-whether another live `OutputGraph` holds the key.
-
-**State of the changelist (2026-09-30, latest).** The trunk **traces**
-(`Dynamo bytecode transform time: 1.08 s`) and the boot dies on the dynamo name-map
-`KeyError: 'in_proj_qkvbfg_a'`. Established about it, in order:
-
-- It is raised **inside the traced `Glm5NextDecoderLayer.forward`** — the frame chain
-  is `profile_run → _dummy_run → execute_model → glm4_1v.py:2358 → the compile wrapper
-  → eval_frame.compile_wrapper → Glm5NextModel.forward:748 (layer(...)) → module
-  _call_impl → Glm5NextDecoderLayer.forward:456 → KeyError`. `TORCHDYNAMO_VERBOSE=1`
-  adds **no** internal stack trace, i.e. it is a plain Python KeyError in the traced
-  frame, raised in a callee dynamo **inlined** (the reported line is the `def`, which
-  is how inlined frames surface).
-- It is **not** raised from any of dynamo's four `nn_modules` lookup sites:
-  `/home/dank/dynamo-probe` (a `sitecustomize.py` on `PYTHONPATH`,
-  `VLLM_DYNAMO_DUMP_ATTRS=1`) wraps three of them and stayed silent across two runs
-  with the probe confirmed installed in both the API-server and worker processes.
-- It is **not** a string-keyed lookup of a literal: `in_proj_qkvbfg_a` appears in the
-  whole tree only as the KDA layer's attribute (`kda.py:217`, used at `:407`) and in
-  the `load_weights` shard table (`model.py:780-785`). The four
-  `proj_name = prefix.split(".")[-1]` sites in the quantization utils do compute
-  exactly this bare key, but every one of them is guarded (`if proj_name not in
-  fused_mapping: return`), so they cannot raise it. The pluggable-layer registry is
-  keyed by class name, not module path.
-- It reproduces on **every** run of this configuration (5 of 5), and it is
-  independent of patches 04/05: with breaks genuinely tolerated, those remove breaks
-  that were never fatal in the first place — the KeyError is the only thing standing
-  between this changelist and a boot.
-
-**Next instrument, precisely:** the inlined caller has to be named. Two cheap ways:
-`TORCH_LOGS=+dynamo` on a boot and grep the log for the key (dynamo prints inlined
-frames with their source), or wrap the inlining candidates in that forward — the mHC
-CustomOps (`hc_pre` / `hc_post` / `hc_fused_post_pre`, `layers/mhc.py`) and the
-indexer op — with a probe that reports on `KeyError`. Until that names the caller, this
-is a dynamo-side name-resolution failure and not a fixable fork bug.
-
-Two verified facts bound the rest of the shape: `suppress_errors` does **not** cover
-graph breaks (only `fullgraph=False` does), and the GLM trunk can never satisfy
-`fullgraph` because its indexer metadata syncs to host
-(`glm5next/amd/sparse_indexer.py:263,419`: `int(...max().item())`) — both are dead
-fallbacks on the decode path that the fork author already replaced, so they are not
-the blockers they appear to be at first glance.
-
-### Two structural constraints on any further fork work
-
-- **No new HIP kernels are possible.** The image ships `r4d.so` as a binary with no
-  sources. The obvious second lever — the 9 % router — is gated
-  `num_experts in {128, 256, 512}` in `router/r4d_route_router.py` and the binary
-  contains only `r4d_moe_route_kernel<128|256|512>` instantiations. GLM has **288**
-  experts, so it falls back to the generic sort-based top-k and **cannot be moved to
-  the fused router without rebuilding libr4d**. Only Python/config/inductor changes
-  (and runtime-JIT Triton) are available.
-- **`--kv-cache-dtype fp8` is a dead end for capacity too.** The plan's KV reserve
-  is dtype-blind (1.76 GiB at 131,072 either way), so fp8 buys no arena and no
-  window; it only removes the only MLA decode path gfx12 has (trap 1 above).
-
-### DFlash2 for GLM — what the drafter actually is
-
-`incoai/GLM-5.3-Flash-DFlash2` is **not** a GLM-shaped drafter and needs no MLA/DSA
-port: its `config.json` declares `architectures: ["DFlash2DraftModel"]` and
-`model_type: "qwen3"` — 5 dense layers, hidden 4096, 32 heads / 8 kv heads,
-head_dim 128, sliding attention (window 2048), `is_causal: false`,
-`use_cache: false`, `dflash_config` = block 8, conv group 16, selector rank 256 /
-top-k 16, and **`target_layer_ids: [5, 14, 24, 33, 42]` against
-`num_target_layers: 45`**.
-
-The fork already maps that architecture:
-`registry.py:636 "DFlash2DraftModel": ("qwen3_dflash2", "DFlash2Qwen3ForCausalLM")`,
-and the spec method is `"dflash"` (`config/speculative.py:67`). So the port is
-**target-side only**: the GLM trunk must expose auxiliary hidden states at those five
-layer indices, the same interface `deepseek_v2.py` (also MLA+MoE) implements —
-`set_aux_hidden_state_layers()` / `get_eagle3_default_aux_hidden_state_layers()` plus
-capture at `idx in self.aux_hidden_state_layers` in the layer loop. `glm5next`
-implements neither.
-
-**The load was tried, and the gap is exactly one interface.** Payload fetched to
-`/home/dank/models/GLM-5.3-Flash-DFlash2` and hash-verified (`model.safetensors`
-2,342,169,800 B, sha256 `b038e1d9…` = the Hub's LFS oid). Booting the engine with
-`--speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":8}'`
-accepts the config and the drafter and dies at worker init with:
-
-```
-RuntimeError: Model does not support EAGLE3 interface
-```
-
-So nothing about the drafter, the method name, or the payload is wrong — the GLM
-trunk simply does not implement the aux-hidden-state contract the dflash runner
-requires. The reference is in this same image, twice: `deepseek_v2.py`
-(`set_aux_hidden_state_layers` / `get_eagle3_default_aux_hidden_state_layers` at :1948-1951,
-capture at :1554 and :1579) and the fork's own `qwen4_exp` trunk. The contract is
-small:
-
-- `self.aux_hidden_state_layers: tuple[int, ...]` on the trunk, default `()`, set
-  by the wrapper's `set_aux_hidden_state_layers()`;
-- in the layer loop, `if idx in self.aux_hidden_state_layers: aux_hidden_states.append(<single-stream state entering layer idx>)`
-  (DeepSeek appends `hidden_states + residual`, with a TP all-gather fixup when the
-  rank holds fewer tokens than `positions`);
-- `Glm5NextModel.forward` returns `(hidden_states, aux_hidden_states)` when the
-  list is non-empty;
-- `get_eagle3_default_aux_hidden_state_layers()` supplies a default triple when the drafter
-  config omits `target_layer_ids` (this drafter provides `[5, 14, 24, 33, 42]`, so
-  the runner derives trunk indices `[6, 15, 25, 34, 43]`).
-
-**The GLM-specific unknown** is what "the single-stream state entering layer k"
-means for a trunk whose layers carry mHC multi-stream state: DeepSeek's
-`(hidden_states, residual)` pair has no counterpart here — GLM's decoder layer
-returns `(hidden_states, residual, post, comb)` with `hc_expand`/`hc_pre` mixing
-`hc_mult` streams, so the plausible answer is the pre-mHC single-stream input `x`
-(4096-wide, matching the drafter's `hidden_size`), not a post-layer value. That is
-a semantic choice to validate by acceptance rate, not by inspection.
-
-**Budget no speedup.** Per the profile above, a verification pass pays pointwise
-cost proportional to its token count — the same reason MTP4 measured flat — and
-MTP4's acceptance was already excellent (3.76-4.90). The port is worth doing as
-*enabling work* (it is the prerequisite for any spec decode on this endpoint, and
-it is small), not as a performance fix, and the honest expectation on this shape is
-flat decode with a prefill regression like MTP's until the pointwise mass is fused.
-
-**The port exists now: `scripts/patches/glm53-dflash-01-aux-hidden-states.patch`** (WIP,
-independent of the compile changelist — it patches the same file as compile-01, so
-compile-01 goes first if both are applied; base `008eeaa2…` → patched `bc9913a1…`). It
-implements exactly the contract read out of this image: `EagleModelMixin` on
-`Glm5NextModel`, capture of the single-stream `hidden_states` ENTERING each requested
-layer in the loop, the conditional `(hidden_states, aux_hidden_states)` tuple return, and
-the three `SupportsEagle3` member attributes plus both interface methods on the
-wrappers (capture shape from `deepseek_v2.py:1551-1585`; tuple consumption at
-`v1/worker/gpu/cudagraph_utils.py:621`).
-
-**The patch went through five versions, and the decisive discovery was where the
-instance actually lives.** The traceback's `set_eagle3_aux_hidden_state_layers(self.model)`
-is called on the model the runner LOADED — and for GLM-5.3-Flash that is
-`Glm5NextForConditionalGeneration` (the multimodal wrapper; the APIServer log names it),
-which holds the text model as `self.language_model = Glm5NextForCausalLM`
-(`glm5next/common/model.py:1113`). Patching only `Glm5NextForCausalLM` (v4) still failed
-with the identical error because the `isinstance` ran against the wrapper. **v5 fixes
-both wrappers** and is the first version to clear the gate (both TP workers log
-`Using Eagle3 auxiliary layers from config: (6, 15, 25, 34, …)`).
-
-The full evolution, each an instructive trap:
-
-1. **v1** hand-wrote the two methods the traceback names → identical error. The methods
-   are not the whole contract.
-2. **v2** inherited `SupportsEagle3` as a base (the image's own idiom — `laguna.py:717`,
-   `mimo_v2.py:834`, …) → dies at class creation with `Cannot create a consistent method
-   resolution order (MRO) for bases …`. GLM's wrappers already carry four/five
-   Protocol-derived bases, and simulation confirmed the ForCausalLM's five-base ordering
-   cannot take `SupportsEagle3` in *any* position.
-3. **v3** declared the members as class attributes (MRO-safe) — but only on
-   `Glm5NextForCausalLM` → identical error (the wrong wrapper, see above).
-4. **v4** added all three members (`supports_eagle3`, `has_own_lm_head`,
-   `has_own_embed_tokens` — a runtime-checkable Protocol's `isinstance` walks ALL
-   `__protocol_attrs__` *including base-Protocol members* via `getattr_static`, verified
-   experimentally in-image with stock typing) on `Glm5NextForCausalLM` → identical error.
-5. **v5** adds the three attributes to BOTH wrappers + the two methods on both
-   (`Glm5NextForCausalLM` unwraps to `self.model` — the `EagleModelMixin` trunk;
-   `Glm5NextForConditionalGeneration` delegates to `self.language_model`, the
-   `pixtral.py:498` pattern for exactly this wrapper shape) → **gate passes**.
-
-**The lesson generalizes twice over**: when a runtime error names an interface, read the
-interface's declaration first — and establish WHICH INSTANCE the check runs against
-before patching anything: multimodal wrappers (`…ForConditionalGeneration`) are loaded
-models, not pass-throughs, and the text model behind `self.language_model` is invisible
-to a check run on the wrapper.
-
-**Retired diagnostic (do not resurrect as-is): the `/probe` startup hook.** An
-`op_probe.py` + `sitecustomize.py` pair was mounted to wrap vLLM module forwards and
-torch's name-map lookups. It produced no evidence and caused a **fork bomb** (see the
-hazard note under experiment-window discipline). Use `TORCH_LOGS` for dynamo questions
-instead — it needs no startup hook.
-
-Note also that the KV-cap patch's predecessor arithmetic in
-[[dankserver-qwen38-fp8-expert-offload-tuning]] should be read the same way: its
-"gather-bound" conclusion came from the same throughput arithmetic that this page's
-profiler window has now falsified.
-
-### The KV-cache grouping journey: clearing the unifier, then the kpool tail
-
-With patch 01 v5 the gate passes and the boot moves to KV-cache layout
-construction — where the next two blockers live, both in
-`v1/core/kv_cache_utils.py`:
-
-1. **The page-size unifier refuses GLM's MLA family.** With a draft model the
-   spec dict gains non-MLA drafter attention specs and `HiddenStateCacheSpec`
-   entries, so `_get_kv_cache_groups_glm5_next` (which requires ALL attention
-   specs to be `MLAAttentionSpec`) bails to the generic grouping path — whose
-   `unify_kv_cache_spec_page_size` raises
-   `NotImplementedError: Layer language_model.model.layers.3.self_attn.indexer.k_cache: page size is not divisible… cannot be padded. Padding is only supported for non-MLA attention layers.`
-   MLA layers cannot be *padded* (sparse MLA indexes whole token rows) — but
-   they CAN grow `block_size` when the max page is an exact multiple. The
-   indexer's page is just `tokens_per_state x` the MLA page, so
-   **`glm53-dflash-02-kv-unify-mla-multiple.patch`** bumps the unified max page
-   to a multiple of every `MLAAttentionSpec` page before the loop: all MLA-family
-   layers then take the legal divisible block-growth branch, drafter + mamba
-   layers pad. The 2026-09-30 window proved it: unify passes, memory profiling
-   passes, torch.compile passes, and the boot reaches cudagraph capture — the
-   NotImplementedError is gone.
-2. **The kpool tail kernel pins its block.** Cudagraph capture then dies in
-   `kpool_ops.kpool_decode_update_and_maybe_write_cache_batched`:
-   `assert tail_kv_cache.shape[2] == pool_size`. The tail cache must view as
-   `[num_blocks, 2, index_kpool, head_dim]` (`KpoolTailSpec(block_size=index_kpool)`,
-   `Glm5NextTailCache` in `glm5next/common/attention.py:159`), but the tail's
-   page (8192 B) *divided* the raised max page, so the unifier's divisible branch
-   grew its `block_size` by the ratio and the view became
-   `[num_blocks, 2, index_kpool x ratio, head_dim]`. Verified not the hybrid
-   promotion: the log carries no "Hybrid KV cache manager is disabled" warning,
-   so `disable_hybrid_kv_cache_manager` is unset and
-   `unify_hybrid_kv_cache_specs` never ran. **`glm53-dflash-03-kv-unify-tailpin.patch`**
-   (= 02 + a `KpoolTailSpec` branch that pads `page_size_padded` instead of ever
-   growing its block) **verified 2026-09-30**: the assert is gone and the boot
-   captures the DFlash2 drafter's FULL cuda graphs on both TP workers
-   (`Capturing model for DFlash2 speculator...`). Supersedes 02 in the repo.
-
-3. **The capacity wall: the generic path cannot alias the mamba states.**
-   Past capture, the boot dies at `_check_enough_kv_cache_memory`:
-   `Available KV cache memory: 0.92 GiB` against `31.76 GiB needed at max len
-   65536` (32.91 GiB at 131,072 — nearly flat, a fixed intercept, not
-   per-token KV). Two coupled causes, read out of the image:
-   - **The glm5 tensor layout aliases the mamba states INTO the MLA pages**
-     (`get_kv_cache_config_from_groups` places every mamba layer at the MLA
-     offsets — that is why GLM-5.3 fits 32 GiB cards at all, and why the
-     expert plan's KV registry is calibrated at 13,600 B/token). On the
-     generic path the mamba groups claim their OWN blocks, so ~30 GiB of
-     unaliased state pages appears.
-   - **The plan's KV reserve (`expert_activation_registry.py:_glm53_kv_bytes_per_token`,
-     a flat 13,600 B/token) knows nothing about the drafter.** The engine caps
-     KV at the plan's reserve, so even a correct generic-path estimate would
-     be starved.
-
-   Learned en route: the engine AUTO-SETS the attention block size to
-   **2,432 tokens** (`interface.py:918: Setting attention block size to 2432
-   tokens to ensure that attention page size is >= mamba page size`), so every
-   attention group's block counts scale from that floor — a 16-token drafter
-   block would need 4x the shared `num_blocks`; with the global 2,432 all
-   groups align.
-
-   **`glm53-dflash-04-kv-groups-draft.patch`** (+ **`-04b-kv-registry-dflash.patch`**
-   for the registry) keeps the aliased glm5 pool and extends it: the grouping
-   partitions the hidden-state specs (`HiddenStateCacheSpec` is an
-   `MLAAttentionSpec` subclass — excluded before the exact-type
-   `type(spec) is MLAAttentionSpec` partition) and the drafter attention
-   specs; the drafter layers form their own uniform group (per-group pages,
-   NO unification — the MLA block size stays at the engine-set 2,432 and the
-   kpool guard `block_size % 512 == 0` is never stressed); the layout returns
-   a 10-tuple with `draft_group`/`hidden_groups`; the builder sizes ONE
-   shared backing allocation (`allocate_kv_cache` asserts all tensor sizes
-   equal) as aliased pool + appended draft/hidden tensors with their own
-   strides; the memory estimator adds the extras. Patch 04b extends the
-   registry: 13,600 + 4,096 (drafter 8 kv heads x 128 x K+V x bf16) +
-   5 x hidden_size x 2 B (the aux hidden-state cache) per token when
-   `speculative_config.use_aux_hidden_state_outputs` is set.
-
-   The pure logic is smoke-tested locally (CPU torch + the image's real spec
-   classes against synthetic GLM+draft+hidden spec sets): grouping shape,
-   layout recognition, builder block math and memory accounting all pass.
-   Real-fit arithmetic: per-token ≈ 58,656 B → reserve ≈ 4.15 GiB at max len
-   65,536 (arena shrinks ~13.5 → ~10.3 GiB, ~1,650 slots), num_blocks ≈ 29 x
-   2,432 tokens ≈ 70,500 ≥ 65,536.
-
-All of these mount over site-packages via the scratch launcher's
-`KVUNIFY_PATCH=1` knob (host `/home/dank/dflash-patch2/new/…`, patched shas
-`5521e50b…` for `kv_cache_utils.py` and `1a623d5c…` for
-`expert_activation_registry.py`; apply-verified against the image's pristine
-files). The window script's decision pipeline was also fixed for a SIGPIPE
-death (`head -80` on a `podman logs` grep pipeline killed one run before
-log-save and restore; the log is now captured to a file FIRST, then grepped).
-
-
-## Measured arms (all 2026-09-30, one boot each)
-
-| arm | decode | prefill | verdict |
-| --- | --- | --- | --- |
-| baseline int4, MAXLEN 131,072, util 0.92, NBATCH 2048 | 10.03 / 10.08 / 9.41 | 877 / 878 / 885 | qualified incumbent |
-| + `--expert-precise-memory` (arena 13.08 → 14.51 GiB) | 9.88 / 9.91 | 918 | **neutral** — proves residency is no longer the binding term |
-| + MTP4 spec decode (draft loads, acceptance excellent) | 9.64-11.16 | **255** | **rejected** — see below |
-| + MTP4 + explicit `--kv-cache-memory-bytes` (patched cap) | boots, 151,767-token KV | — | enabler for the MTP arm only |
-
-### Why MTP is rejected here (it is not that it fails)
-
-MTP **works** on GLM in this fork: the head loads, the architecture resolves to
-`Glm5NextMTPModel`, and acceptance is excellent —
-
-```
-SpecDecoding metrics: Mean acceptance length: 3.83-4.90,
-Per-position acceptance rate: 0.917/0.833/0.583/0.500 … 0.962/0.846/0.692/0.615,
-Avg Draft acceptance rate: 69-97%
-```
-
-— and it buys **nothing**: decode 9.6-11.2 tok/s against 10.0 without it, while
-prefill **collapses 918 → 255 tok/s**. The reason is structural to offload: a
-verification forward gathers the **union** of the batch's experts, so spec
-decode amortizes compute but not a residency-bound gather. It also costs 2.6 GiB
-of arena. On a compute-bound (fully-resident) shape this would invert; here it
-does not.
-
-## Served surface (verified 2026-09-30 against the live engine)
-
-| capability | state | evidence |
-| --- | --- | --- |
-| `/v1/models` | `glm53-flash`, max_model_len 131,072 | model list probe |
-| completion | yes | `bench_glm.py` runs above |
-| **tool calls** | **yes** — needs `--enable-auto-tool-choice --tool-call-parser glm47` | a `get_weather` prompt returns `tool_calls: [{"id": "chatcmpl-tool-…", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Seattle\"}"}}]`, `finish_reason: tool_calls` |
-| **vision** | **yes** | a 64x64 solid-red PNG is answered "The image appears to be a solid red color…" |
-| reasoning channel | **yes** — needs `--reasoning-parser glm47` | without it the thinking leaks into `content` (measured: content began with the answer, then "Let me verify: … = 391", then a bare `</think`) with an empty reasoning field; with it, a train-time prompt answers in `content` ("**Step 1: Add the hours.** 14:20 + 3 hours = 17:20 …") and the chain of thought lands in **`reasoning`** — the vLLM field name the fleet documents (llama.cpp endpoints use `reasoning_content`) |
-
-Without the tool flags the engine answers **HTTP 400** — `"auto" tool choice
-requires --enable-auto-tool-choice and --tool-call-parser to be set` — which
-breaks every tool-using agent on the endpoint. Both parsers are `glm47`, the
-newest GLM dialect the fork ships (`glm45`/`glm47` exist for both).
-
-## Fork traps found (for the engine authors)
-
-1. **`--kv-cache-dtype fp8` is illegal on gfx12 with this model — and it buys
-   nothing even if fixed.** The `ROCM_AITER_MLA_SPARSE` backend's
-   `_use_rocm_sparse_triton()` requires a non-fp8 KV cache, and that rope-free
-   Triton path is the **only** MLA decode available on gfx12: aiter's MLA decode
-   op is registered under `@if_aiter_supported`, which is gated on **CDNA > 2**
-   (`is_aiter_found_and_supported()` → `get_cdna_version() > 2`). With fp8 KV
-   the MTP draft step demands an op that cannot exist here:
-   `RuntimeError: Worker failed with error ''_OpNamespace' 'vllm' object has no
-   attribute 'rocm_aiter_mla_decode_fwd''`. KV dtype therefore stays auto/bf16.
-   (`VLLM_ROCM_USE_AITER_MLA=1` does **not** help — the umbrella gate is the
-   CDNA check, not the env var.)
-   - **The capacity payoff is measured at zero.** The point of fp8 KV is fitting
-     more tokens in the same reserve, and this fork's reserve model is
-     **dtype-blind**: the fp8 boot printed the *identical* plan line as bf16 —
-     `KV reserve 1.76 GiB (13600 B/token x 131072 + mamba pages)`. The reserve is
-     dominated by the per-sequence KDA state (34 linear-attention layers) and the
-     DSA indexer's pools, not by the MLA KV bytes those 13,600 B/token barely
-     cover — so halving the KV dtype cannot move it.
-   - **What a fix would take, if it were ever wanted**: a new fp8 path in the
-     rope-free Triton kernels (`rocm_sparse_attn_prefill` / `rocm_sparse_attn_decode`
-     in `v1/attention/ops/rocm_aiter_mla_sparse.py`) that dequantizes pages
-     in-kernel, plus loosening `_use_rocm_sparse_triton()`. The fork's
-     `_r4d_fp8_paged_mqa_logits` shows the author already has an fp8 MQA-logits
-     path, but the sparse-MLA backend is not wired to it. Verdict 2026-09-30:
-     **do not spend effort here** — the payoff is capacity, and the reserve is
-     dtype-blind.
-2. **The plan's KV reserve under-models this model, and the cap enforces the
-   mistake.** The plan charges 13,600 B/token + a small mamba term, but GLM
-   carries a per-sequence KDA state (34 linear-attention layers) and, under MTP,
-   the draft's KV. `gpu_worker` then *caps* KV at that reserve, so a legal
-   `--max-model-len` becomes unbootable:
-   `ValueError: To serve at least one request with the model's max seq len
-   (131072), (2.07 GiB KV cache is needed, which is larger than the available KV
-   cache memory (1.74 GiB))`. Raising `max_model_len` only helps past ~440k
-   tokens (the cap's slope is ~13,845 B/token vs the need's ~12,700 + 0.52 GiB
-   fixed) — where the reserve would eat a third of the arena.
-   `scripts/patches/gpu_worker-kv-cap.patch` lets an explicit operator KV
-   request raise the cap; **it is not part of the deployed shape**, which boots
-   on the plan's own reserve.
-3. **A window is forced, not chosen.** The production `max_model_len 131072` is
-   what the reserve model requires for this payload (below ~92k the boot
-   refuses); it is not a capacity decision.
-4. **Expert prefetch remains poison** — unchanged from the qwen campaign
-   (36% precision, +40-51% decode from deleting it). No GLM arm re-opened it.
-
-## Operations
-
-```bash
-# rollback to the pre-cutover qwen FP8 + MTP4 shape. The trio is in git; the
-# qwen FP8 payload is NOT on disk any more, so re-fetch it first. The R9V state
-# dirs are still on the host (/home/dank/r9v-state/):
-git show 2339fae -- scripts/ | git apply   # or: git checkout 2339fae -- scripts/
-./scripts/deploy.sh dankserver --service frank-archon
-
-# payload inventory (34 shards + 2 extras, weight-map index) — in the GLM era
-# this was scripts/download-frank-primary.sh (REPO=Intel/GLM-5.3-Flash-W4A16-AutoRound,
-# EXPECTED_SHARDS=34, git show f2bcf84). The same-named script today fetches
-# Qwen/Qwen3.8-Flash-Next-FP8 (scripts/download-frank-archon.sh:21):
-bash scripts/download-frank-archon.sh --verify-only
-
-# live telemetry (needs the profiler window; with_stack=false is mandatory —
-# with stack the stop path hangs and the trace buffer grows to tens of GB):
-podman logs frank-archon-glm 2>&1 | grep "expert_lru.py"
-```
-
-**Experiment-window discipline (learned the hard way, 2026-09-30).** This engine
-**ignores SIGTERM**: after `systemctl stop` its container sits in `podman ps` state
-`Stopping` for 20+ minutes *while still answering `/health` with 200*. That makes a
-liveness probe a lying oracle for a restore — twice a restore script printed
-`PROD HEALTHY` while `systemctl is-active` said `failed` and the runtime mask was
-still in place. The restore oracle must be a **conjunction**: `is-active` ==
-`active` **and** `is-enabled` == `enabled` **and** no `/run/systemd/system/<unit>`
-mask symlink **and** the container is owned by the unit and not `Stopping`. The
-repair recipe, all four steps needed:
-
-```bash
-sudo mv /run/systemd/system/frank-archon.service /tmp/stale-mask-N.service  # runtime mask SURVIVES stop and blocks start
-sudo systemctl daemon-reload
-sudo systemctl reset-failed frank-archon.service                            # a failed unit refuses to start
-podman rm -f frank-archon-glm                                               # SIGKILL; `podman stop` hangs on the same SIGTERM
-sudo systemctl start frank-archon.service                                   # then ~7 min to health 200
-```
-
-Also: **never let two experiment jobs overlap.** A later job's `podman rm -f
-glm-probe` cleanup ran while an earlier job was still mid-flight and deleted its
-container, so one whole round produced no evidence at all (`no container with name
-glm-probe found`) — check `proc://` for running jobs before opening a window.
-
-**Never put a diagnostic hook on `PYTHONPATH` without a marker guard — it is a fork
-bomb.** On 2026-09-30 a `sitecustomize.py` mounted at `/probe` (to wrap dynamo's
-name-map lookups and vLLM module forwards) produced a self-replicating chain of
-`python3` processes 19+ levels deep inside the probe container: `sitecustomize` is
-imported by *every* interpreter, the hook imported torch/vLLM at startup, something in
-that import chain spawned a helper python, and the child inherited `PYTHONPATH` (hence
-the hook) and spawned again. It was still alive 18 minutes later with the container
-still showing `Up`. Symptoms: `ps -eo pid,ppid,comm --forest` shows a deep ladder of
-identical interpreters, each child's PPID being the previous one. Guard (now in the
-hook): arm once per environment via a marker in `os.environ`, which children inherit —
-
-```python
-if os.environ.get("VLLM_PROBE_ARMED") == "1":
-    pass                       # an ancestor already armed it; stay inert
-else:
-    os.environ["VLLM_PROBE_ARMED"] = "1"
-    <install probes>
-```
-
-Kill recipe: `podman rm -f <experiment-container>` (the container cgroup owns every
-generation), then verify with a precise count over `/proc/<pid>/environ` for the marker
-variable — **never** pattern-kill an interpreter name, because the media stack on this
-host (romm, jellyfin, plex, glances) runs python in production.
-
-```bash
-# boot: ~7 min warm page cache, longer cold; health returns 503 while loading
-curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health
-
-# experiment windows (production stopped + `systemctl mask --runtime`, which
-# self-clears at reboot) ran from the host-side scratch launcher
-# /home/dank/serve_glm_autoround.sh: env knobs MAXLEN SEQS NBATCH UTIL KVBYTES
-# KVDTYPE SPEC SPECJSON_FILE TOOLPARSER REASONINGPARSER TELEM GCHUNKS GLANES
-# PROFDIR GLM_PATCH_MOUNT COMPILE_PATCH GATHER_PATCH DRAFT_MOUNT ENVARGS.
-# DRAFT_MOUNT=1 mounts /home/dank/models/GLM-5.3-Flash-DFlash2 at /draft;
-# SPECJSON_FILE passes a whole --speculative-config JSON (JSON with no spaces,
-# word-split by the launcher's unquoted $EXTRA).
-bash /home/dank/serve_glm_autoround.sh
-```
-
-Boot gate: the engine refuses a legal-looking shape in two measured ways — the
-KV cap above, and the profile-run activation peak (`TritonMLA` indexer scores:
-64 heads x 2,048 topk x NBATCH tokens x 2 B = 2.06 GiB at NBATCH 8192), which is
-what killed the first two boots at util 0.95.
-
-`PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` is part of the qualified
-shape, not a tuning: the plan fills VRAM to its modelled headroom by design, so
-the default allocator's fragmentation fails the profile/graph-capture phase
-(measured on the deployed unit: "Tried to allocate 662.00 MiB. GPU 0 … 240.00
-MiB is free").
-
-The `deploy.sh` e2e gate also drives the admin panel's open-links spec, which
-walks every guarded vhost — including `http://192.168.0.112:8989/` on
-FrankBook-Pro. When that Mac mini is asleep the spec fails on a `page.goto`
-timeout and the deploy reports `DEPLOY INCOMPLETE` even though the engine is
-healthy; wake the host (magic packet to `de:6b:6d:c1:52:23`) or re-run
-`tests/e2e/run.sh` once it is up. Seen 2026-09-30: 32 passed, this one failed,
-and the WoL packet did not wake it (macOS "Wake for network access" needs
-toggling on the mini itself).
-
-## Rollback targets
-
-| state | where |
-| --- | --- |
-| qwen FP8 + MTP4 (2026-09-29 → 09-30) | commit `2339fae`; payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` was deleted from the host (2026-10-18: `/home/dank/models/` has `StillDeadcode`, `rad`, `tpl.jinja`) — rollback = re-fetch |
-| R9V radiance Qwen3.8-Flash-Next Q8_0 | [[frank-archon-r9v-radiance-qwen38-q8]], state dirs `/home/dank/r9v-state/{q8xl,uncensored}` |

serverAppConfigs-docs/LLM/frank-archon-llamacpp-qwen38-flash-next-q8.md

@@ -1,270 +0,0 @@
----
-type: configuration
-title: frank-archon llama.cpp Qwen3.8-Flash-Next Q8_0 Runbook
-description: RETIRED 2026-09-26 → 09-27 frank-archon runbook — unsloth llama.cpp (the b11030-mix prebuilt, ROCm gfx120X, bundled runtime, root systemd unit) serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 (175.3 GiB across six split parts) on Dankserver's two Radeon AI PRO R9700s (gfx1201), with the shared Q8_0 MTP drafter (depth 2) chained behind ngram-mod, BF16 vision, and the routed experts + 51B-param n-gram embedding table computed in host RAM. Port 7770, one stream, full 262,144 ctx. Cutover 2026-09-26, replacing the same-day Lucebox DS4.1 deployment; superseded 2026-09-27, and frank-archon runs [[frank-archon-radiance-qwen38-iq4r]] today (2026-10-18).
-tags:
-  - frank-archon
-  - dankserver
-  - llamacpp
-  - unsloth
-  - qwen38-flash-next
-  - q8_0
-  - mtp
-  - ngram-mod
-  - spec-decode
-  - vision
-  - rocm
-  - gfx1201
-  - r9700
-timestamp: 2026-09-26
----
-
-# frank-archon llama.cpp Qwen3.8-Flash-Next Q8_0 Runbook
-
-The dankserver LLM endpoint (:7770, alias **frank-archon**) as it stood from
-2026-09-26 to 2026-09-27 (retired; the port today serves
-[[frank-archon-radiance-qwen38-iq4r]]):
-unsloth llama.cpp serving `unsloth/Qwen3.8-Flash-Next-GGUF` at **Q_8**
-(the 6-part `Q8_0/` main, 175.30 GiB) on **both** Radeon AI PRO R9700s,
-with the **shared Q_8 MTP drafter** (`draft-mtp`, depth 2) layered with
-**`ngram-mod`**, and **vision** through `mmproj-BF16.gguf`. It replaces the
-Lucebox DS4.1 quadlet that held the port for hours earlier the same day
-([[frank-archon-lucebox-ds4]]), which itself replaced
-[[frank-medium-radiance-mxfp4]].
-
-## Why this shape
-
-- **One runtime for the fleet.** All three endpoints are llama.cpp now —
-  one metric namespace (`llamacpp:*`), one reasoning contract
-  (`reasoning_content`), one tool contract (`--jinja`). The two-day
-  experiment with a single-vendor engine (lucebox) ended because Qwen
-  Flash-Next at Q8 on llama.cpp delivers better quality than the DS4.1
-  fp2/fp3 stream, plus vision, plus the full 262k window, at comparable
-  speed.
-- **The placement law, not a preference:** the routed experts and the
-  51B-param n-gram (PLE) embedding table are **computed in host RAM**
-  (`-ot exps=CPU,per_layer_token_embd.weight=CPU`) and never streamed over PCIe. The
-  x4-pair measurement on the same model shape: streaming loses (3.7 t/s),
-  computing in RAM does not care (20+ t/s). The dense weights, attention,
-  projector and MTP head sit on the two cards (~29 GiB); the ~100 GiB of
-  expert pages live in mmap page cache — `--no-mmap` is forbidden (it
-  forces the table into anonymous memory; the cgroup kill is silent) and
-  pinning the PLE table with a GPU target cost 23 → 13.4 t/s when tried.
-  The tensor names are verified, not guessed: the GGUF v3 header dump
-  (`dump-gguf.py`, run on the fetched parts 2026-09-26) shows the PLE
-  table is one tensor — `per_layer_token_embd.weight`, dims
-  `[160, 320001536]`, Q8_0, 54.4 GiB, exactly the whole of part 3 — and
-  the model is `qwen4exp` with 512 experts top-10
-  (`qwen4exp.ple.ngram_size=3`, `heads_per_ngram=8`). The same tensor is
-  the MTP head's borrow target, so the override is required even if the
-  placement were optional.
-- **Two cards, layer split** (`-ts 1,1`): both R9700s are on CPU root
-  ports with full PCIe atomics (the ProArt B850-Creator swap,
-  2026-09-20), but this deployment uses llama.cpp's layer split —
-  `-ot` tensor overrides do not compose with tensor-parallel mode, and
-  `exps=CPU` is the override the placement law needs. The 27B-dense
-  equivalent on this host measured 1991 t/s prefill / 18.8 t/s decode in
-  this shape; Flash-Next Q8 with CPU experts lands in the same class
-  (measured numbers below).
-- **Spec chain: `draft-mtp,ngram-mod`.** MTP is a property of the FILE:
-  the shared-Q8_0 head (`mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf`,
-  2,786,568,256 B) is loaded via `--model-draft` and borrows the target's
-  embedding tensors — which is why `per_layer_token_embd.weight` must be
-  CPU-resident in the override set (a GPU-resident borrow crashes the
-  draft load with `invalid vector subscript`). `ngram-mod` is draftless,
-  so it chains legally behind the drafter. Depth 2: each longer
-  verification round re-batches the CPU expert reads, so depth buys less
-  than bandwidth here. **The drafter must sit on ROCm1** — the head also
-  borrows the target's `output.weight`, which a `-ts 1,1` split places on
-  the last-layers card; with tensor overrides set the draft scheduler runs
-  `n_copies==1`, and a draft context elsewhere aborts at load
-  (`ggml-backend.cpp:941`, "pre-allocated tensor (output.weight) in a
-  buffer (ROCm1) that cannot run the operation"). The 2026-09-26 cutover
-  crash-looped 4× on `ROCm0` before the probe confirmed `ROCm1` loads —
-  the frank-medium precedent (drafter on the last-layers card) was right.
-- **Vision, explicitly.** This build never autoloads projectors — without
-  `--mmproj` the server comes up healthy and text-only and answers image
-  requests with 500 (the frank-medium 2026-09-17 trap). The BF16 projector
-  (907,542,944 B) is named in the serve script.
-
-## Deployment
-
-|Piece|Value|
-|---|---|
-|Unit|`scripts/frank-archon.service` — root systemd unit, `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity`, relinked to `/etc/systemd/system/`|
-|Serve script|`scripts/serve-frank-archon.sh` — validates binary + all 6 parts + drafter + mmproj before exec|
-|Env tunables|`scripts/frank-archon.env` — paths and numbers only; bind is `BIND_HOST` (never a bare `HOST`: HIP reads it)|
-|Runtime|`/home/dank/GitPackages/unsloth-llama-b11030/llama-server` — unsloth b11030-mix prebuilt (commit 6ba30d05b), ROCm gfx120X, bundled runtime via `LD_LIBRARY_PATH`|
-|Payload|`/home/dank/models/Qwen3.8-Flash-Next-GGUF/` — `Q8_0/` (6 parts), `MTP/mtp-...-shared-Q8_0.gguf`, `mmproj-BF16.gguf`; fetched by `scripts/download-frank-archon.sh` (sha256 + byte-count verified)|
-|Port / alias|7770 / `frank-archon` — unchanged from the lucebox and radiance eras; no client moves|
-|Context|262,144 × 1 slot (`PARALLEL_SLOTS=1`, one stream, the model's full native window)|
-|Sampling|Greedy server default (deterministic e2e); Qwen-recommended sampling rides `models.yml` `compat.extraBody` per request|
-|Template|The GGUF's embedded template via `--jinja`; reasoning → `reasoning_content` (`--reasoning-format deepseek`)|
-|Observability|`frank-archon` Prometheus job at `192.168.0.235:7770` — cross-host LAN scrape (the netavark published-port quirk that forces `host.containers.internal` is same-host only); exports `llamacpp:*`|
-
-### Cutover mechanics
-
-The deployer (`deploy_llm_archon` in `scripts/deploy-remote.sh`) gates on
-every artifact at its declared byte count (`stat -c %s`, never `wc -c` —
-which read the whole 377 GiB lucebox file for minutes on 2026-09-26),
-retires the lucebox user quadlet to dated home trash
-(`~/.AGENTTRASH/20260926-llamacpp-cutover/`), waits for port 7770 to be
-truly free, relinks + enables + restarts the root unit, and waits up to
-1800 s for `/health` (cold start page-ins ~175 GiB and builds the graphs).
-
-### Preflight the GGUF without downloading it
-
-GGUF metadata lives in the header and HF honours ranged GETs, so a few MB
-answers every sizing question before a 13 GB transfer
-(`Range: bytes=0-8000000`). Parse: magic `GGUF`, u32 version, u64
-n_tensors, u64 n_kv, then key/value pairs (string = u64 len + bytes; ARRAY
-= u32 elem type + u64 count; BOOL is 1 byte — the type hand-written parsers
-forget). Walk arrays by byte size even for a summary: a naive parser
-desynchronises on `tokenizer.ggml.tokens` and lands gigabytes off. Read:
-`general.architecture`, `<arch>.block_count`, `<arch>.context_length`,
-`<arch>.nextn_predict_layers` (MTP is a property of the FILE — the `-mtp`
-build carries `blk.<n>.nextn.*`; `--spec-type draft-mtp` then loads the
-head against the MAIN model's weights, and `--model-draft` would load the
-file twice), `<arch>.head_count_kv`, `<arch>.key_length`/`value_length`,
-`<arch>.full_attention_interval`, `<arch>.ssm.state_size`/`group_count`/
-`time_step_rank`, and for the projector `clip.projector_type` +
-`clip.has_vision_encoder`.
-
-KV arithmetic for the hybrid (16 × (3 Gated DeltaNet + 1 Gated Attention)):
-only full-attention layers take KV
-(`attn_layers × n_head_kv × head_dim × 2 × bytes_per_elem`; f16 = 2 B,
-q8_0 = 1.0625 B, q4_0 = 0.5625 B); DeltaNet state is a fixed per-slot
-buffer (~0.16 GiB/sequence at 48 layers/128 d_state/48 v-heads f32); the
-MTP draft context costs ≈ 4 KiB/token. `--parallel N` DIVIDES `--ctx-size`
-between slots — it does not multiply.
-
-## Verification (2026-09-26 deploy)
-
-- `/v1/models` reports `frank-archon`.
-- Chat completion returns `content` + `reasoning_content`.
-- Tool call returns well-formed `tool_calls` (`--jinja`).
-- Real image request answered from the projector (not a health check —
-  health cannot see vision).
-- Spec chain live: `draft acceptance` lines in the unit journal; ngram-mod
-  has no startup banner (identify behaviourally: `draft_n` > 0 where text
-  repeats).
-- `reasoning_effort: high` → 200 (the template's `{xhigh, medium, low}`
-  gate is folded by `reasoningEffortMap` in `models.yml`).
-- VRAM/RAM headroom: `rocm-smi` per card + `free -g` after the first long
-  completion.
-
-_**Measured (2026-09-26 battery, 6/6 pass).** Load: ~7 s warm (mmap from
-page cache; the fetch's own verify pass pre-warms it — a cold first load
-pages in 175 GiB). Decode: **11.8 t/s** `predicted_per_second` at
-depth-0 prose, 128-token generation, draft_n 109 / accepted 72 (0.66 on
-reasoning-heavy prose; **0.864 mean len 2.73** on the tool-call task).
-Vision: the probe image (green square centered on a red field) described
-exactly. VRAM: GPU0 29% / GPU1 36% of 32 GiB (~9.3 / ~11.5 GiB — the
-output-head card carries the drafter, as designed; ~22 GiB free per
-card). RAM: ~11 GiB used, ~226 GiB page cache (the payload itself),
-available ~235 of 246 GiB. `reasoning_effort: high` → 200. Two cutover
-traps for the record: a LAST-ACK tail from a LAN client held
-`wait_port_free` until it cleared (a foreign process on the port gates
-the deploy correctly — but the deployer's `/health` wait cannot tell WHO
-answers; after any aborted deploy, confirm the responder is the unit's
-own MainPID), and llama-server's SIGTERM cleanup hung after
-spec-decode service (logged `cleaning up before exit` and never exited —
-`kill -9` was needed; `systemctl stop` relies on TimeoutStopSec)._
-**KV law (from the -lv 4 load log, cross-checked by the GGUF header):**
-29,696 B/token total ctx-scaled VRAM — 24,576 B main KV (only 12 of 48
-layers carry attention, `full_attention_interval=4`; 2 KV heads x 256,
-f16) + 3,072 B indexer + 2,048 B draft KV. Everything else (weights
-~6.7 GiB, recurrent state 0.33 GiB, compute buffers ~4.1 GiB) is
-ctx-independent. At 262,144: 7.25 GiB scaled, 18.75 GiB total, 9.37
-GiB/card. At 100,000: 2.77 GiB scaled, 14.3 GiB total, 7.1 GiB/card.
-`--parallel N` splits the same pool — slots do not multiply KV.
-
-**Reshape (later the same day): four full slots, V q8_0, hot expert
-blocks on-GPU.** `CONTEXT_SIZE=1048576` / `PARALLEL_SLOTS=4` — four
-conversations, each the FULL native 262,144 window (per-slot window
-unchanged, so no client or models.yml implication; slots divide the
-pool, never multiply it). `CACHE_TYPE_V=q8_0` with K held at f16: main
-KV drops 24,576 → 18,816 B/token (−23.4%), pool law 23,936 B/token.
-The shared MTP drafter's KV ignores the V type (measured: draft pool
-2,048 MiB = the f16 law at 1,048,576 cells — one layer's worth, the
-miss costs ~0.5 GiB on ROCm1). At 4×262,144 the pool is ~23 GiB across
-both cards: main 9,408 MiB + indexer 1,536 MiB per card, draft
-2,048 MiB on ROCm1.
-
-Expert blocks stay GPU-resident — and the measurement below shows it
-cannot matter *which* ones. `-ot` granularity is the whole
-per-block expert tensor (512 experts share one tensor), so "critical
-experts" = blocks co-located on the card that holds their layers:
-blocks 0-4 on ROCm0, 45-47 on ROCm1 (measured 2.42/2.49 GiB per
-block). The placement law that makes it work: b11030 wants ONE
-comma-joined `--override-tensor` (repeats log DEPRECATED), and the
-loader applies the FIRST matching pattern per tensor — the
-block-specific patterns MUST precede the broad `exps=CPU` sweep, or
-the sweep silently swallows them (verified live: broad-first left
-every expert on CPU with no warning anywhere in the log).
-
-Measured headroom, deployed fill **5/3** (blocks 0-4 on ROCm0, 45-47
-on ROCm1 — the operator directed the fill past the 3-4 GB checkpoint
-floor the same day, "shift a few more experts"): card0 2.53 GiB free
-at boot, 2.23 GiB under load with the vision graph exercised; card1
-3.31 GiB at boot, 3.15 GiB under load. Card1 cannot take another
-block — 3.31 − 2.49 = 0.82 GiB, and the vision graph (~0.43 GiB) plus
-boot drift alone exceed that — so 5/3 is the physical limit at the
-4×262144 / V-q8_0 shape. Earlier 4/3 state for reference: card0 5.02
-GiB boot free. Live numbers (4/3 era): content decode 13.05 t/s
-(acceptance 0.58, mean len 2.16), reasoning phase 7.24 t/s
-(acceptance 0.74, mean len 2.48), two concurrent full generations both
-delivered, prompt cache reusing 42/64 tokens across requests. All
-three Prometheus fleet targets up; the 5/3 deploy re-verified chat,
-vision and the target on the same battery class.
-
-**Which blocks, measured (same day):** asked whether the `-ot` regex
-could target the *most active* blocks, I instrumented the live server
-instead of guessing — `scripts/frank-archon-expert-activity.py`
-samples llama-server with AMD IBS (`perf record -e ibs_op// --data`)
-while 4 mixed concurrent generations run, maps each sampled load
-address through `/proc/<pid>/maps` back to its backing tensor (extents
-derived from consecutive GGUF header offsets — never from a quant
-type-size table, whose errors silently mis-bucket sample tails), and
-aggregates per block and per expert. Verdict, 75 s window / 2.7M
-expert-tensor samples: **per-block read volume is FLAT, 0.93×–1.08× of
-mean across all 40 CPU-resident blocks** (0-4 and 45-47 absent —
-GPU-resident). MoE top-k routing fires every block on every token, so
-"most active layers" do not exist at `-ot` granularity: the regex can
-express any 8-block set, and any set is equivalent to any other — the
-lever is COUNT, already at the 5/3 VRAM ceiling. The real skew is
-per-EXPERT (top 10% of the 48×512 (block,expert) cells carry 40.5% of
-samples; hottest cell 0.1%), below the reach of a tensor-level
-override. Not a benchmark: it instruments production serving in place
-at negligible IBS overhead; re-measure any future checkpoint or
-traffic mix with `frank-archon-expert-activity.py <pid>` (or
-`--analyze-only` against the saved sample file in `/tmp`).
-
-## Rollback
-
-The lucebox DS4.1 image `localhost/lucebox-rocm:gfx1201` is still in dank's
-rootless storage; the 378 GiB payload is not (Dankserver's `/home/dank/models/`
-holds only `StillDeadcode`, `rad` and `tpl.jinja` as of 2026-10-18), and the
-quadlet-era repo files were deleted by `8500600` rather than trashed — there is
-no `dankserver/.AGENTTRASH/20260926-llamacpp-cutover/` (see
-[[frank-archon-lucebox-ds4]]). Restore = recover the `.container` from
-`git show 8500600^`, stop the root unit (`systemctl stop frank-archon`; the
-fleet law forbids `systemctl disable` on these hosts — it removes the unit
-entry, which here is the symlink into the checkout), `daemon-reload`, start the
-user quadlet. The Qwen payload (~179 GiB) is no longer on disk either, so a
-forward re-swap is a re-fetch plus a redeploy.
-
-## Fleet context
-
-- [[frank-dragoon-unsloth-qwen38-27b-q8xl]] — the era's Dankclaw :7770, the same
-  b11030-mix binary family on gfx1151 (retired 2026-09-28; frank-dragoon runs
-  [[frank-dragoon-flash-server-v2-native]] today).
-- [[frank-zealot-gemma4-12b]] — the era's Dankclaw :7771, mainline llama.cpp on
-  the 3090, and then the fleet's only audio-capable endpoint. Both are history:
-  frank-zealot serves [[frank-zealot-llama-prism-bonsai2-pq20]] today and no
-  fleet endpoint takes audio (the audio path left with Gemma4 on 2026-10-01).
-- The harness contract lives in `/home/blank/.omp/agent/models.yml`. The era's
-  `llamacpp (dankserver)` numbers (`contextWindow 262144`, `maxTokens 65536`,
-  `input: [text, image]`) are long gone — that provider reads
-  `contextWindow: 200000`, `maxTokens: 16384`, `input: [text, image]` as of
-  2026-10-18.

serverAppConfigs-docs/LLM/frank-archon-lucebox-ds4.md

@@ -1,222 +0,0 @@
----
-type: configuration
-title: frank-archon Lucebox DeepSeek V4 Runbook
-description: RETIRED 2026-09-26 frank-archon runbook (active for hours on 2026-09-26) — the Lucebox runtime (github.com/Luce-Org/lucebox, local image localhost/lucebox-rocm:gfx1201) serving DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, 43-layer MoE, 256 routed experts, top-6) on Dankserver's R9700 with the routed experts computed on CPU from the mmapped page cache. Replaced the same day by [[frank-archon-llamacpp-qwen38-flash-next-q8]] (llama.cpp Qwen3.8-Flash-Next Q8_0); the image is still in rootless storage but the payload is gone from the host, and frank-archon runs [[frank-archon-radiance-qwen38-iq4r]] today (2026-10-18).
-tags:
-  - frank-archon
-  - dankserver
-  - lucebox
-  - deepseek-v4
-  - ds4
-  - rocmfp23
-  - gfx1201
-  - r9700
-  - quadlet
-  - moe
-timestamp: 2026-09-26
----
-
-# frank-archon Lucebox DeepSeek V4 Runbook
-
-> [!warning] RETIRED 2026-09-26 (same day it went live)
-> Replaced by [[frank-archon-llamacpp-qwen38-flash-next-q8]] — unsloth
-> llama.cpp serving Qwen3.8-Flash-Next Q8_0 on both R9700s. This runtime
-> served for hours; the image `localhost/lucebox-rocm:gfx1201` is still in
-> dank's rootless storage, but the 378 GiB payload was deleted from the host
-> (`ls /home/dank/models/` on Dankserver, 2026-10-18: `StillDeadcode`, `rad`,
-> `tpl.jinja` only), and the quadlet-era repo files were deleted by `8500600`
-> (2026-09-26) — they survive only in git history, not in a `.AGENTTRASH`
-> directory (`dankserver/.AGENTTRASH/` holds only `20260916-vllm-radiance`).
-
-The dankserver LLM endpoint (:7770, alias **frank-archon**) as it stood on
-2026-09-26: the
-[Lucebox](https://github.com/Luce-Org/lucebox) runtime serving
-`Lucebox/DeepSeek-V4.1-Flash-ROCMFP23-GGUF` (377.2 GiB, 405,349,859,328
-bytes, LFS oid
-`2daea6ca1715bc987aae201e63445d3f7abdc2337c05f5f569f5e50237ce8be2`) on a
-single Radeon AI PRO R9700 (gfx1201). It replaced
-[[frank-medium-radiance-mxfp4]] (vllm-radiance TP=2), retired earlier the
-same day.
-
-## Why this shape
-
-DeepSeek-V4.1-Flash is a 43-layer MoE: 256 routed experts + 1 shared expert
-per layer, top-6 routing. The ROCMFP23 quantization is fp2 gate/up with fp3
-down experts (Q2_0_ROCMFP2 type 107 + Q3_0_ROCMFPX type 104). At 377.2 GiB
-the model fits neither the 63.7 GiB of combined VRAM nor the 246 GiB of RAM,
-so it is served the way llama.cpp serves big models by default: the GGUF is
-mmapped, pages pull from disk/page cache on demand, dense work (attention,
-shared expert, embeddings, output projection) and the MLA KV cache run on
-the R9700, and the routed experts are computed on CPU. The 246 GiB RAM
-(upgraded for this deployment) holds the working set of active experts; the
-rest streams from the 1.8 T NVMe.
-
-Lucebox's `--ds4-fused-decode` requests a monolithic GPU load; on a model
-this size that allocation fails and the backend falls back to **hybrid
-expert placement + layered decode** — that fallback is the intended serving
-mode here, not an error.
-
-## Image
-
-`localhost/lucebox-rocm:gfx1201`, built on the host from
-`github.com/Luce-Org/lucebox` `Dockerfile.rocm`:
-
-```
-git clone --recurse-submodules https://github.com/Luce-Org/lucebox.git
-cd lucebox
-# Three local modifications (tracked at the time as
-# dankserver/frank-primary/lucebox-gfx1201.patch, deleted by 8500600):
-#   1. Dockerfile.rocm drops the ROCm *-dev apt packages (the 7.14.1-full
-#      base already ships them in /opt/rocm; the apt repo does not serve
-#      them) and the three runtime math libs.
-#   2. Dockerfile.rocm installs the official cmake 3.31.6 binary (jammy's
-#      3.22 predates the `block` command ROCm 7.14 hipblaslt-config.cmake
-#      requires, CMake >= 3.25).
-#   3. ggml-cuda.cu: `ggml_cuda_node_launches_nothing` moved above the
-#      `#ifdef USE_CUDA_GRAPH` guard — `ggml_cuda_node_is_noop` calls it
-#      unconditionally, but the macro is never defined by the build
-#      system, so the in-guard definition was dead and the use failed.
-git apply ../lucebox-gfx1201.patch
-podman build --build-arg LUCE_HIP_ARCHES=gfx1201 \
-             --build-arg ROCM_VERSION=7.14.1-full \
-             -t localhost/lucebox-rocm:gfx1201 -f Dockerfile.rocm .
-```
-
-- The prebuilt `ghcr.io/luce-org/lucebox-hub:rocm` is **not usable as-is**:
-  it is built for gfx1151 + ROCm 6.4.1, and gfx1200/gfx1201 are not
-  code-object compatible.
-- `ROCM_VERSION=7.14.1-full` matches the host driver's ROCm 7.x line (the
-  radiance image's core-7.14 userspace runs on this host). A 6.4.x userspace
-  on a 7.x driver can segfault at model load. Only `-full` tags exist for
-  7.14.x.
-- The binary lands at `/opt/lucebox-hub/server/build/luce_server`.
-
-## Model
-
-`/home/dank/models/DeepSeek-V4.1-Flash/DeepSeek-V4.1-Flash-ROCMFP23.gguf`,
-fetched with `huggingface_hub` (the xet-native path — plain `curl` full GETs
-400 against HF's xethub CDN for this repo; see the fetch note in
-`scripts/hf-fetch.sh`). Byte count and sha256 are deploy-gated in
-`deploy_llm_archon`.
-
-## Deployed flags (2026-09-26)
-
-From `dankserver/frank-primary/frank-primary-entrypoint.sh` (deleted by `8500600`;
-recover it with `git show 8500600^:dankserver/frank-primary/frank-primary-entrypoint.sh`) — the unit
-bypasses the image's entrypoint, which only forwards a fixed Qwen/DFlash
-flag set and cannot express the DS4 flags):
-
-| Setting | Value |
-|---|---|
-| Model | `DeepSeek-V4.1-Flash-ROCMFP23.gguf` (377.2 GiB) |
-| Device | `--target-device hip:0` (`HIP_VISIBLE_DEVICES=0`; the Raphael iGPU at index 2 stays out) |
-| Window | `--max-ctx 131072` (the qualified Strix Halo window; DS4's MLA KV is compressed, so the pool is small) |
-| Chunk | `--chunk 8192` (the qualified profile's chunk) |
-| KV dtype | `--cache-type-k q4_0 --cache-type-v q4_0` (the qualified launch) |
-| Decode | `--ds4-fused-decode` (falls back to hybrid expert placement on this host) |
-| Prefill | `--ds4-prefill sparse` (the batched prefill; `exact` runs at token-wise decode cost) |
-| Routing | `--ds4-expert-top-k 6` (the model default; do NOT reduce on adaptive qtype-105/106 artifacts — measured degenerate output at top-4, DS4.md 2026-08-05) |
-| Draft | none — no DSpark draft is published for V4.1-Flash; plain autoregressive decode |
-| Identity | `--model-name frank-archon`, `--host 0.0.0.0 --port 7770` |
-
-## API surface
-
-OpenAI Chat Completions (streaming + tool metadata), `/v1/models`,
-`/health`, OpenAI Responses, Anthropic Messages, and Open WebUI model
-metadata — the fleet contract is unchanged from frank-medium.
-
-## Cutover procedure
-
-1. Image built and model fetched (gates in `deploy_llm_archon`).
-2. `deploy.sh` → `deploy_llm_archon`: stops and **masks** the radiance
-   unit (`frank-medium`), waits for :7770 to free, installs
-   `dankserver/frank-primary/quadlet/frank-primary.container` (also deleted by
-   `8500600`; recover it from `git show 8500600^`), starts
-   `frank-archon.service`, waits for health (1800 s start period — first
-   start page-ins the dense weights and builds the graphs).
-3. Verify: `/health` 200 is not enough — a real chat completion, a tool
-   call, and an Open WebUI completion.
-4. Retire the radiance unit to the home trash (never `rm`); the MXFP6
-   checkpoint payload did not stay — it is gone from the host (see Rollback).
-
-## Rollback
-
-Unmask and start the radiance unit (`systemctl --user unmask frank-medium &&
-systemctl --user start frank-medium`) and stop/mask frank-archon. The
-`localhost/radiance-vllm:0.13.0-e060895` image is still in dank's rootless
-storage; the `/models/Qwen3.8-27B-PARO-MXFP6` checkpoint is not — Dankserver's
-`/home/dank/models/` holds only `StillDeadcode`, `rad` and `tpl.jinja` as of
-2026-10-18, so this rollback needs a re-fetch of the payload.
-
-## Verification (2026-09-26)
-
-_Filled in after the cutover: health, real completion, tool call, Open WebUI
-completion, measured prefill/decode, `systemctl --user is-enabled/is-active`
-state._
-
-## Dual-R9700 follow-up (2026-09-28 probe era)
-
-After this single-card runbook was retired, the DS41 engine got a dual-R9700
-attempt on the `feat/ds41-engine` branch (`main` rejects the checkpoint with
-`unsupported model architecture 'deepseek41'`). Findings that outlive the
-probe:
-
-**Budget both cards explicitly, or you waste half the box.** The engine
-sizes the streamed-expert cache LAST, after the model, so uncapped it absorbs
-every spare byte on the stream device — symptom: card1 at 33.0/34.2 GB
-looking busy while card0 sits at 14.2/34.2 GB; the busy card is holding a
-cache of things still streaming from NVMe. Pin all four:
-
-| env | card | meaning |
-|---|---|---|
-| `LUCE_EXPERT_BUDGET_MB` | hip:0 | primary hot experts (~11.96 MiB each) |
-| `LUCE_EXPERT_SECONDARY_BUDGET_MB` | hip:1 | secondary experts (needs `LUCE_DS4_MOE_TP*`) |
-| `LUCE_EXPERT_STREAM_CACHE_MB` | hip:1 | cap the auto-sized stream cache |
-| `LUCE_EXPERT_STREAM_DEVICE=1` | hip:1 | which card holds the slots and computes them |
-
-Working full-card shape: 21000 + 20000 + 8000 → card0 ~28.6/31.86 GiB,
-card1 ~27.3/31.86 GiB, resident experts 3,360 of 15,360.
-
-**Expert TP DOES work on gfx1201+gfx1201.** A retired quadlet recorded
-`LUCE_DS4_MOE_TP_INPROC` as "qualified for gfx1151+gfx1201, not
-gfx1201+gfx1201" — that is WRONG; peer access negotiates and the secondary
-tier comes up clean: `[deepseek4-moe-tp] enabled mode=in-process local=hip:0
-secondary=hip:1`. It is the only lever that materially raises resident
-expert count, and resident count is the entire bottleneck. Confirm every run
-with `expert ownership:` and `hybrid experts ready:` lines.
-
-**The placement file must live under the mounted models dir.** JSON
-`{"owners": [...]}`, 15,360 entries indexed `layer*384+expert`, values
-`primary`/`secondary`/`stream`. It is a ranking, not a promise: each device
-keeps what fits its budget and demotes the rest. Anywhere outside
-`/models/...` gives `cannot open expert placement` and a SILENT single-tier
-fallback — a silent-wrongness failure, not a crash.
-
-**V4.1 refuses layer split and layer-major prefill** — only `--ds4-prefill
-exact` (token-by-token) is legal on a 32 GiB card and `--ds4-fused-decode`
-is refused there. With ~22% of experts resident each token pulls ~187 cold
-experts × ~12 MiB; if `/proc/<pid>/io` `read_bytes` climbs into terabytes
-the model is larger than the box's RAM and you are in page-cache thrash at
-the NVMe ceiling — a capacity verdict, not a tuning problem. Low GPU util
-with a pegged link is the EXPECTED signature of PCIe-expert-stream-bound
-decode, not a broken run.
-
-**Harness traps, each of which produced a wrong number:** never `env -i` the
-podman call (rootless podman runs as `dank`; `env -i HOME=/root` makes
-PODMAN try `/root/.config` and die `stat /root/.config: permission denied`
-before the container exists — pass container env with `-e`); never put a
-large prompt on curl's argv (`Argument list too long` at ~128 KB — use
-`--data-binary @file`); size prompts by growing the file until `wc -c`
-reaches the byte target AND assert depth against the engine's own
-`usage.prompt_tokens` (nominal 4 chars/token was really 5.73; a chunk loop
-that appends a fixed ~152-byte line per nominal 200-byte chunk undershot
-30%); the server runs as the container's PID 1, so tear down with
-`podman rm -f`, never host `pkill`; drop page cache between arms
-(`sync; echo 3 > /proc/sys/vm/drop_caches`) or the first arm reads warm and
-every later one is a lie.
-
-Read the placement log — it is the whole diagnosis
-(`arch=deepseek41 layers=40 embd=5120 experts=384 used=6 ff=2304
-hash_layers=0` — hash_layers=0 is correct for V4.1; `engram: layers=[1,14]
-… 2 table(s) embedded (not mapped)` — the 189.13 GiB Engram tables are never
-mapped, so they do not evict the expert page cache).

serverAppConfigs-docs/LLM/frank-archon-r9v-radiance-qwen38-q8.md

@@ -1,296 +0,0 @@
----
-title: frank-archon — R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0
-service: frank-archon
-host: dankserver
-port: 7770
-status: retired
-deployed: 2026-09-27
-supersedes: "[[frank-archon-r9v-radiance-qwen38-uncensored]]"
-tags: [llm, frank-archon, r9v, vllm, radiance, gfx1201, q8]
----
-
-# frank-archon — R9V radiance-vLLM Q8_0 (dankserver, RETIRED 2026-09-27 → 09-30)
-
-The Dankserver LLM endpoint **was** the **R9V radiance-vLLM fork** (retired
-2026-09-30, when [[frank-archon-glm53-flash-w4a16]] took the port; the engine
-live today, 2026-10-18, is radiance with
-[[frank-archon-radiance-qwen38-iq4r]])
-(`drwolfen/radiance-vllm-qwen4exp`, docker-pinned prebuilt image bundle
-`2dac17a2`) serving the **unsloth Qwen3.8-Flash-Next-GGUF Q8_0** payload
-(175.30 GiB, 6 GGUF shards, sha256-verified per part) with the blockfp8
-MTP4 head and the mmproj, on **both R9700s** (gfx1201, BDFs `0000:03:00.0`
-/ `0000:07:00.0`) as `frank-primary` on **:7770** (the `frank-archon` wire alias
-arrived with the 2026-10-09 rename).
-
-Cut over from the uncensored IQ4_XS payload on 2026-09-27 at Frank's
-direction. Rollback target at the time:
-[[frank-archon-r9v-radiance-qwen38-uncensored]] — its state dir
-(`/home/dank/r9v-state/uncensored/`, with `setup.json` and
-`restore-config-20260927.env`) is still on the host as of 2026-10-18, but the
-IQ4_XS payload itself is not (`/home/dank/models/` holds only `StillDeadcode`,
-`rad`, `tpl.jinja`), so that flip now needs a re-fetch.
-
-## Why the swap
-
-The uncensored IQ4_XS payload measured 59–96 tok/s decode with
-draft-acceptance swings; the Q8_0 payload trades VRAM-resident expert
-headroom for precision — quantization to Q8_0 doubles expert bytes per
-block, so the tiered-expert residency budget halves — the probe-era
-11 tok/s decode was residency-bound (expert streaming, not compute).
-The "Known costs" section below carries the accepted trade.
-
-## Production shape (repo-tracked)
-
-The repo carries a **thin** wiring; the engine shape lives on-host:
-
-| Piece | File | Law |
-|---|---|---|
-| Unit | `scripts/frank-archon.service` | Root unit, `User=dank`, `LimitMEMLOCK=infinity`, `ExecStop=podman stop -t 120 frank-archon-r9v`, `Restart=on-failure` |
-| Env | `scripts/frank-archon.env` | Production overrides ONLY: bind 0.0.0.0, :7770, alias `frank-archon`, container `frank-archon-r9v`, GPU BDFs, dev-overlay adapter, `R9V_FULL_MUTABLE_CACHE=0`, `R9V_PLE_READY_TIMEOUT=1800`, prefix caching ON, sharp chat template |
-| Serve | `scripts/serve-frank-archon.sh` | Layers setup state under repo env → execs the fork's own `scripts/launch.sh` via the docker→podman shim → `docker wait` |
-| Deployer | `scripts/deploy-remote.sh` `deploy_llm_archon` | Gates on setup state ready, image digest, PLE bytes, all six shard files + mmproj; MainPID false-green check |
-
-**Why the tuned shape lives on-host:** the q8 profile's setup run
-qualified the `R9V_*` config (tiered-expert placement, per-rank offload
-budgets, 80-slot tiered cache, PLE **pinned** residency 54.4 GB, MTP4,
-parser wiring, `MAX_MODEL_LEN=131072`). It is written to
-`/home/dank/r9v-state/q8xl/setup.json` and the serve script layers it
-UNDER the repo env (repo keys win). Copying those values into the repo
-would drift from the qualification — don't.
-
-## Spec decode: MTP4 only — the ngram-mod ask is NOT expressible
-
-llama.cpp chains drafter types (`--spec-type draft-mtp,ngram-mod` on
-frank-zealot/large). The fork's `speculative_config` is a **single dict**
-(one method per engine): `mtp`, `ngram`, `ngram_gpu`, `dflash`, … are
-mutually exclusive — no chaining seam exists in the fork. What IS active:
-
-- **MTP4** (`R9V_MTP_SPEC_TOKENS=4`, blockfp8 draft head, draft TP=2) —
-  measured mean acceptance length 3.19–5.00 during qualification.
-- **The PLE n-gram** — architectural (`ngram_embedding` module), rides
-  along automatically; it is not the llama.cpp `ngram-mod` matcher and
-  cannot be "added" as a second spec method.
-- `ngram_gpu` exists as an ALTERNATIVE single method (fork's GPU n-gram
-  proposer) — swapping MTP4 for it is an either/or, not a chain.
-
-## Sharp chat template (fleet parity with frank-dragoon)
-
-`scripts/qwen-sharp-v22.4.0.jinja` is served via the launcher's opt-in
-`R9V_CHAT_TEMPLATE_FILE` passthrough (fork commit `30d4a32`): the file is
-mounted read-only at `/opt/r9v/share/chat-template.jinja` and passed as
-`--chat-template`, which wins over the tokenizer dir's
-`metadata/chat_template.jinja`; `R9V_CHAT_TEMPLATE_KWARGS` carries
-`--default-chat-template-kwargs {"preserve_thinking":true}` (the template
-defaults `preserve_thinking` true natively; request-level kwargs still win
-per request). The template folds reasoning efforts (high/xhigh/max →
-xhigh, minimal/low → low), matching `models.yml`'s `reasoningEffortMap`.
-The sha-pinned `metadata/chat_template.jinja` in the package is NEVER
-edited in place — mutating it drifts the qualification.
-
-## Two q8-only runtime laws
-
-1. **Quantized inject shard** — the Q8_0 payload quantizes BOTH fused
-   hyper-connection shards (the IQ4_XS package ships inject as F32), so
-   the shard-uniformity check rejects the fused projection at load.
-   `R9V_DEV_QWEN4_EXP_ADAPTER_PY` (fork commit `363d606`) mounts a
-   dev-overlay adapter that dequantizes a quantized inject shard
-   in-stream; uncensored payloads pass through unchanged.
-2. **`R9V_FULL_MUTABLE_CACHE=0`** — the runtime overlay's `env.always`
-   injects the uncensored 160-slot full-mutable cache geometry, which the
-   q8 profile's 80-slot tiered cache fails with a baseline mismatch. The
-   launcher passes this key AFTER the overlay envs (docker last-wins).
-
-## Fork checkout + shim (on-host, outside the repo)
-
-- Checkout: `/home/dank/GitPackages/radiance-vllm-qwen4exp` (fork commits
-  `363d606` dev-overlay, `2fe6dbb`, `50a79ec` qualification doc,
-  `30d4a32` chat-template passthrough, `2de0357` PLE-timeout forwarding —
-  local-only, remote is GitHub)
-- State: `/home/dank/r9v-state/q8xl/setup.json` (0600, `ready: true`)
-- Model dir: `/home/dank/models/Qwen3.8-Flash-Next-Q8_0-R9V/` — 175.30 GiB
-  payload as six `Q8_0/Qwen3.8-Flash-Next-Q8_0-0000N-of-00006.gguf` shards
-  + mmproj; PLE derived at the pinned size 54,400,261,120 B
-- Docker→podman shim: `/home/dank/GitPackages/radiance-vllm-qwen4exp/.podman-shim/docker`
-  (PATH-prepended by the serve script). The R9V tooling is docker-pinned;
-  the shim carries three laws:
-  1. **Image ID law** — the bundle is a containerd/OCI docker-save; docker
-     keys the image by its OCI MANIFEST digest (`sha256:2dac17a2…`), podman
-     by config digest but preserves the manifest digest in **RepoDigests**.
-     The shim answers `image inspect sha256:X` from RepoDigests and rewrites
-     bare `sha256:X` run refs to `repo@sha256:X`.
-  2. **LogConfig law** — podman inspect answers
-     `HostConfig.LogConfig.Config: null` where docker returns the log-opts
-     map (the doctor crashes on `.get("Config", {}) == null`). The shim
-     records `run --log-opt` pairs and normalizes inspect output.
-  3. **docker info law** — podman lacks `SecurityOptions`/`ServerVersion`;
-     the shim answers rootless + `27.5.1` (liveness probe).
-- venv: `~/.r9v-venv` (huggingface_hub for fetch/verify)
-
-## Bring-up battery (qualification, 2026-09-27)
-
-- Setup `ready: true`; all 20 payload artifacts size+sha256 verified
-  (175.30 GiB total across the six shards).
-- Vision: 64×64 red PNG → "Red" (79 tok, 7.4 s end-to-end on the scratch
-  port). Prose: 100 tok in 9.1 s ≈ 11.0 tok/s wall **on a cold cache with
-  reasoning tokens included in the wall clock** — decode-rate context, not
-  a bench (no throughput bench was run, per Frank's direction).
-- MTP4: mean acceptance length 3.19–5.00 across the battery.
-- PLE pinned proof: `mapping_bytes = mapping_rss_bytes =
-  54,400,261,120` (fully RSS-resident); doubling for re-pin would need
-  108,800,522,240 ≤ available 140,876,808,192; registration took 446 s.
-
-## Memory shape (measured)
-
-- Load: 21.84 GiB / worker at ~158 s; cold start → serve ~16 min.
-- PLE pinned residency 54.4 GB in host RAM (vs the uncensored payload's
-  ssd-mmap residency).
-- Context envelope: `R9V_MAX_MODEL_LEN=131072`; harness `models.yml`
-  carries `contextWindow: 131072`, `maxTokens: 36864` (compaction-gap
-  math: 70% × 131,072 = 91,750 + 36,864 = 128,614 ≤ 131,072). The KV
-  envelope at 131,072 is smaller than the uncensored 196,608 shape.
-- Prefix caching OFF at cutover (`R9V_ENABLE_PREFIX_CACHING=0`), CACHE192
-  off — superseded 2026-09-28: prefix caching is ON (see bench matrix
-  below).
-
-## Prefill / decode bench matrix (2026-09-28, scratch port 7773)
-
-Production stopped for the window (fleet bench law); one matrix plus three
-follow-up runs, fork's own bench-pp + decode_speed harness, corpus-sliced
-prompts with the fork's canonical decode rounds. Baseline E0 = production
-shape (chunk 4096, 80-slot sync cache, MTP4):
-
-| Config | Override | PP8K tok/s | PP131K tok/s | Decode | Verdict |
-|---|---|---|---|---|---|
-| E0 | — | 954.6 | 883.0 | 141.9 ms/step ≈ 23.8 t/s eff | baseline |
-| E1R | chunk 8192 | **1201.4 (+25.9%)** | 844.6 (−4.4%) | 158.5 ms (−8.5% t/s) | short-prompt win, decode tax |
-| E2 | chunk 16384 | — | — | — | boots; 33.3/34.2 GiB VRAM (0.93 free) → requests hang/OOM |
-| E1/E3 | chunk 8192/32768 | — | — | — | PLE registration blew the 600 s engine timeout (see law below) |
-| E4 | cache async=1 | — | — | — | doctor: cache-prepare backend requires synchronous single-fill LRU |
-| E5/E6 | 96/112 slots | — | — | — | doctor: capacities (48, 64, 80) only; expert-budget [108, 198] rank maxima — **80 is the policy max, prod is at it** |
-| E7 | MTP depth 6 | 843.1 | 819.6 | 454–489 ms/step (3.7x worse) | depth 4 stays |
-
-Laws from the matrix:
-
-- **4096 is the prefill-chunk ceiling twice over**: the engine caps
-  scheduled tokens by the speculative config, and E2's VRAM (0.93 GiB free
-  at 16384) shows bigger chunks OOM at request time. 883–955 tok/s is what
-  this payload shape delivers at long context.
-- **Unqualified shapes fail mid-request, not at health** — E2 and E7 both
-  answered /health 200 and then died streaming.
-- **PLE registration can exceed the 600 s engine timeout**
-  (`VLLM_PLE_OFFLOAD_READY_TIMEOUT`) under page-cache churn — measured 740 s
-  after a bench teardown; a production boot failed on it the same morning
-  (unit "Deactivated successfully", no error in journal — the serve script
-  exits 0 on the wait). Fixed with `R9V_PLE_READY_TIMEOUT=1800` in
-  `frank-archon.env` (forwarded by launch.sh `2de0357`).
-
-Prefix caching (E8/E9, `R9V_ENABLE_PREFIX_CACHING=1`, same port): **zero
-VRAM cost** (hits reuse freed KV blocks; load VRAM identical to E0),
-cold path unchanged (first request ≈ cold bench), reuse hits large — 32k
-prompt TTFT 44.2 s → **3.6–4.3 s** (9x), 131k 439 s → **6.9 s** (63x).
-Hit pattern: rounds 1–2 miss, round 3+ hit (identical-prompt and
-conversation-growth probes both). For the harness's resend-every-turn
-traffic that is a ~40–440 s TTFT win per steady-state turn; the one-turn
-miss costs nothing extra. **DEPLOYED 2026-09-28** (Frank's go-ahead): the
-repo env sets `R9V_ENABLE_PREFIX_CACHING=1` (commit `41e9189`), verified
-on :7770 (`--enable-prefix-caching` in the container argv, health 200,
-live completion answering).
-
-## Authoring a new profile (the qualification pipeline)
-
-Order matters at every step:
-
-1. **Package manifest** `packages/models/<pkg>/package.json` — target shards
-   + aux artifacts with sha256. Verify every target's byte count AND sha256
-   after fetch (`curl -fL -C -` per shard; never trust a fetch without `-f`).
-2. **Placement bootstrap** from GGUF headers: parse all shards' tensor
-   tables (a KV-only shard1 is normal for split parts; each part carries its
-   own tensor table). Per-expert full per layer =
-   (`ffn_gate_exps` + `ffn_up_exps` + `ffn_down_exps`)/512; rank slices
-   rank0 = full×416//640, rank1 = full−rank0. Hot counts =
-   floor(measured_budget / per-expert total). **Anchor budgets to the
-   MEASURED hot_bytes of the qualified sibling profile, never invented.**
-3. **Trim + validate**: `tools/trim_experts.py --hot-counts N,M`, then
-   `expert_budget.validate_manifest(m)` +
-   `expert_budget.expert_memory(m, slots, {cache_rank}, runtime=<runtime.json>)`
-   (runtime/catalog are keyword-only). Declared hot_bytes must match
-   computed or it raises.
-4. **PLE derive** (containerized `tools/prepare_ple.py` with
-   `--expected-bytes/--expected-type/--expected-shape`); setup skips
-   re-derivation when the file exists with matching size.
-5. **profile.env**: set values BEFORE `source dual-r9700/profile.env`
-   (set-first-wins; an empty string does NOT survive `:=`).
-
-The canonical bootstrap.json wrapper schema is exactly
-`{schema: "r9v.placement.v1", id, model_package, hardware, artifact_path,
-artifact_sha256, scope}` with `hardware: "dual-r9700-32gb-asymmetric"`; the
-wrapper `id` MUST equal the id in the profile.json placement descriptor
-(setup rejects mismatches: "does not match profile placement"). Never
-fabricate qualification metadata — use
-`scope: "Unranked routing bootstrap only; does not certify fit or headroom"`.
-
-GGUF parsing laws for the bootstrap: expert tensor names are
-`blk.0.ffn_up_exps.weight` — split on "." gives FOUR parts, layer number is
-parts[1] (a 3-part assumption crashes `int("")`); outer-KV value-type sizes
-`{0:1,1:1,2:2,3:2,4:4,5:4,6:4,7:1,8:str,9:arr,10:8,11:8,12:8}`; tensor byte
-extents from consecutive OFFSETS, never a type-size table (mis-buckets tails
-silently); the PLE Q8_0 tensor on this arch is shape (160, 320001536),
-54,400,261,120 B, an entire shard.
-
-Setup/start: `./r9v setup <alias> --state-dir <dir> --model-dir <dir> --image
-sha256:<digest> --local-image --reuse-from <sibling-model-dir>
---accept-model-license` (aux artifacts reuse byte-identical files; F16 vs
-Q8 mmproj will not reuse). Start requires the SAME `--state-dir`; fixed
-placement via `R9V_EXPERT_MANIFEST_PATH` skips headroom/catalog/seed raises.
-The host-only doctor FAILs when another instance holds the cards (it wants
-3 GiB free per rank; production at ~30 GiB/card is a hard wall) — a second
-instance CANNOT coexist, so qualification requires a stop+mask window for
-the incumbent, then unmask+restart.
-
-**Qualification gate (unqualified → qualified):** first start runs the
-workload battery (PONG / tool_calls / vision / long-ctx), the PLE pinning
-proof (`VmLck` ≈ PLE bytes in `/proc/<container-pid>/status` when
-RESIDENCY_MODE=pinned), and a VRAM headroom snapshot (floor 3.3 GiB). If
-headroom is violated, lower hot counts via re-trim and re-derive
-MAX_EFFECTIVE. Only then record state `ready: true` and author the runbook.
-
-Remote-execution gotchas from the same windows: `nohup X &` inside an ssh
-command races ssh teardown — the script can die before writing a byte; run
-long remote scripts synchronously, or setsid + disown, or via a system
-unit. zsh login shells abort on unquoted `===` separators; grep globs need
-quoting through ssh+zsh layers.
-
-## Ops
-
-- **Restart**: `sudo systemctl restart frank-archon` (root unit; the serve
-  script re-gates everything, relaunch through the shim, waits).
-- **First start after boot** re-warms ~175 GiB of page cache; expect
-  minutes to /health; `TimeoutStartSec=1800`.
-- **Scratch/probe**: NEVER probe against the prod state dir. Copy it, set
-  `R9V_HOST_PORT` to a free port and `R9V_CONTAINER_NAME` to a scratch
-  name, and stop the probe before any deploy — a probe holding :7770
-  produces a false-green deploy (the 2026-09-26 lesson, guarded by the
-  MainPID check).
-- **Rollback to uncensored**: point
-  `scripts/frank-archon.env` `R9V_STATE_FILE` back at
-  `/home/dank/r9v-state/uncensored/setup.json`, drop the two q8-only keys
-  (adapter, FULL_MUTABLE_CACHE) and the template keys, redeploy — the
-  uncensored state dir is intact on disk; the IQ4_XS payload is not (it was
-  deleted in the 2026-10-17 cleanup), so this rollback re-fetches first.
-- **Models on the wire**: served alias `frank-primary` (the StarCraft wire
-  names arrived 2026-10-09, `7c77326`); thinking channel is `reasoning` (vLLM);
-  tool parser `qwen3_coder`.
-
-## Known costs (accepted at cutover)
-
-- **Residency-halved experts** — Q8_0 doubles expert bytes vs IQ4_XS, so
-  the tiered cache holds half as many experts in VRAM at the same budget;
-  the probe's 11 tok/s decode was residency-bound, and the production
-  number depends on the re-qualified budgets in the q8 state.
-- **128K envelope** — down from the uncensored 196,608 (which itself was
-  down from 262,144 on llama.cpp).
-- **Reference quant** — back to the unsloth Q8_0 weights (no
-  abliteration); answer-quality profile differs from the uncensored
-  incumbent by design.
-- **Docker-pinned fork** — updates flow through image bundles + the shim,
-  never a bare container run; the launcher stays the single argv owner.

serverAppConfigs-docs/LLM/frank-archon-r9v-radiance-qwen38-uncensored.md

@@ -1,138 +0,0 @@
----
-title: frank-archon — R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS
-service: frank-archon
-host: dankserver
-port: 7770
-status: retired
-retired: 2026-09-27
-superseded_by: "[[frank-archon-r9v-radiance-qwen38-q8]]"
-supersedes: "[[frank-archon-llamacpp-qwen38-flash-next-q8]]"
-tags: [llm, frank-archon, r9v, vllm, radiance, gfx1201, retired]
----
-
-# frank-archon — R9V radiance-vLLM (dankserver, RETIRED)
-
-> **RETIRED 2026-09-27** — superseded the same day by the Q8_0 cutover,
-> [[frank-archon-r9v-radiance-qwen38-q8]] (itself retired 2026-09-30; the live
-> frank-archon runbook today, 2026-10-18, is
-> [[frank-archon-radiance-qwen38-iq4r]]). The uncensored state dir and frozen
-> config are still on disk (`/home/dank/r9v-state/uncensored/` — `setup.json`,
-> `restore-config-20260927.env`); the IQ4_XS **payload is not** — it was deleted
-> in the 2026-10-17 cleanup. The rollback procedure lives in the Q8_0 page's Ops
-> section, with a re-fetch as step one.
-
-The Dankserver LLM endpoint **was** the **R9V radiance-vLLM fork**
-(`drwolfen/radiance-vllm-qwen4exp`, docker-pinned prebuilt) serving
-`Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS` — the abliterated
-Qwen3.8-Flash-Next at IQ4_XS (92.39 GiB, 3 GGUF shards) with the blockfp8
-MTP4 head and the F16 mmproj — on **both R9700s** (gfx1201, BDFs
-`0000:03:00.0` / `0000:07:00.0`) as `frank-primary` on **:7770** (the
-`frank-archon` wire alias arrived 2026-10-09).
-
-Replaced llama.cpp b11030 (Qwen3.8-Flash-Next Q_8) on 2026-09-27 at Frank's
-direction; downtime window ~1 h 20 min (killed 02:47, healthy 04:11 PDT).
-Rollback target: [[frank-archon-llamacpp-qwen38-flash-next-q8]] — the
-llama.cpp trio (unit/env/serve) is at commit `4992e5a`+ (the repo-side
-wiring; host unit was the relinked one); the GGUF payload is no longer on
-disk (Dankserver's `/home/dank/models/` holds only `StillDeadcode`, `rad`,
-`tpl.jinja` as of 2026-10-18), so that rollback is a re-fetch plus a redeploy.
-
-## Production shape (repo-tracked)
-
-The repo carries a **thin** wiring; the engine shape lives on-host:
-
-| Piece | File | Law |
-|---|---|---|
-| Unit | `scripts/frank-archon.service` | Root unit, `User=dank`, `LimitMEMLOCK=infinity`, `ExecStop=podman stop -t 120 frank-archon-r9v`, `Restart=on-failure` |
-| Env | `scripts/frank-archon.env` | Production overrides ONLY: bind 0.0.0.0, :7770, alias `frank-primary` (the wire id became `frank-archon` on 2026-10-09), container `frank-archon-r9v`, GPU BDFs |
-| Serve | `scripts/serve-frank-archon.sh` | Layers setup state under repo env → execs the fork's own `scripts/launch.sh` via the docker→podman shim → `docker wait` |
-| Deployer | `scripts/deploy-remote.sh` `deploy_llm_archon` | Gates on setup state ready, image digest, PLE bytes, shard files; MainPID false-green check |
-
-**Why the tuned shape lives on-host:** the profile's setup run qualified a
-104-key `R9V_*` config (tiered-expert placement, per-rank offload budgets
-146.25/78.75 GiB, CED on, MTP4, parser wiring, `MAX_MODEL_LEN=196608`). It
-is written to `/home/dank/r9v-state/uncensored/setup.json` and the serve
-script layers it UNDER the repo env (repo keys win). Copying those values
-into the repo would drift from the qualification — don't.
-
-**Self-healing law:** the launcher `--detach`es; the serve script therefore
-blocks on `docker wait`. Without the wait, a crashed engine is a green
-unit. `docker wait` cannot stop the container child, hence `ExecStop`.
-
-## Fork checkout + shim (on-host, outside the repo)
-
-- Checkout: `/home/dank/GitPackages/radiance-vllm-qwen4exp`
-- State: `/home/dank/r9v-state/uncensored/setup.json` (0600, `ready: true`)
-- Model dir: `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS/`
-  (92.39 GiB payload + `r9v-data/per_layer_token_embd.iq4_nl.bin` 26.82 GiB
-  PLE, sha256 `34fa36f8…`)
-- Docker→podman shim: `/home/dank/GitPackages/radiance-vllm-qwen4exp/.podman-shim/docker`
-  (PATH-prepended by the serve script). The R9V tooling is docker-pinned;
-  the shim carries three laws:
-  1. **Image ID law** — the bundle is a containerd/OCI docker-save; docker
-     keys the image by its OCI MANIFEST digest (`sha256:2dac17a2…`), podman
-     by config digest but preserves the manifest digest in **RepoDigests**.
-     The shim answers `image inspect sha256:X` from RepoDigests and rewrites
-     bare `sha256:X` run refs to `repo@sha256:X`.
-  2. **LogConfig law** — podman inspect answers
-     `HostConfig.LogConfig.Config: null` where docker returns the log-opts
-     map (the doctor crashes on `.get("Config", {}) == null`). The shim
-     records `run --log-opt` pairs and normalizes inspect output.
-  3. **docker info law** — podman lacks `SecurityOptions`/`ServerVersion`;
-     the shim answers rootless + `27.5.1` (liveness probe).
-- venv: `~/.r9v-venv` (huggingface_hub for fetch/verify)
-
-## Verified at cutover (2026-09-27)
-
-- Setup: **PASS=28 WARN=3 FAIL=0** — 20/20 artifacts size+sha256 verified
-  (~93 GiB), PLE derived at exact pinned size, CED projector verified.
-- First-start doctor battery: **PASS=34 WARN=8 FAIL=0**, including
-  `runtime-worker-transport`: **pinned UVA reads, H2D copies, TP
-  all-reduce verified inside serving workers** on both BDFs.
-- Wire: `/v1/models` → `frank-archon`; `/health` + `/metrics` 200
-  (Prometheus scraping); "Say PONG" (non-thinking) → `'PONG'`, stop;
-  thinking requests populate `reasoning` (vLLM channel).
-- Memlock: the full tiered-expert load ran to healthy **under a plain dank
-  session** before the unit existed — the pin path does NOT hard-require
-  `LimitMEMLOCK=infinity` on this host, but the root unit keeps fleet
-  parity and headroom for the pinned reserve.
-- e2e gate: 32 passed / 1 failed — the failure is `11-admin open links`,
-  which needs FrankBook-Pro (192.168.0.112) awake; the machine was down and
-  WOL from both hosts failed. Suite minus that spec: **31 passed** (the
-  one-time 12-crawl flake re-passes in isolation).
-
-## Memory shape (measured)
-
-- Both cards ~29–30 GiB used of 31.86; engine budget
-  `CPU_OFFLOAD_GB_BY_DEVICE=146.25,78.75` (UVA host pin ≈ 43 GiB on rank 0
-  + 16 GiB PLE pinned reserve; PLE residency mode `ssd` mmap+trim).
-- `MemAvailable` floor: 56.3 GiB (`R9V_MIN_HOST_AVAILABLE_BYTES`).
-- Context envelope: 192,608 native window served as `MAX_MODEL_LEN=196608`;
-  harness `models.yml` carries `contextWindow: 196608`, `maxTokens: 57344`
-  (compaction-gap math: 70% × 196,608 = 137,625 + 57,344 ≤ 196,608).
-
-## Ops
-
-- **Restart**: `sudo systemctl restart frank-archon` (root unit; the serve
-  script re-gates everything, relaunch through the shim, waits).
-- **First start after boot** re-warms ~93 GiB of page cache; expect several
-  minutes to /health; `TimeoutStartSec=1800`.
-- **Scratch/probe**: NEVER probe against the prod state dir. Copy it, set
-  `R9V_HOST_PORT` to a free port and `R9V_CONTAINER_NAME` to a scratch
-  name, and stop the probe before any deploy — a probe holding :7770
-  produces a false-green deploy (the 2026-09-26 lesson, now guarded by the
-  MainPID check).
-- **Qualification artifacts** (kept on-host):
-  `/home/dank/r9v-state/uncensored/qualification-*`,
-  failure bundles `failure-*` (review before sharing — raw logs).
-- **Models on the wire**: served alias `frank-archon`; thinking channel is
-  `reasoning` (vLLM), tool parser `qwen3_coder` (not qwen3_xml like the
-  vllm030 wiring had planned).
-
-## Known costs (accepted at selection)
-
-- **Abliterated weights** (Dyluhn uncensored build) — not the reference
-  model; answer-quality deltas are possible.
-- **192K envelope** — down from 262,144 on llama.cpp.
-- **Docker-pinned fork** — updates flow through image bundles + the shim,
-  never a bare container run; the launcher stays the single argv owner.

serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-iq4r.md

@@ -1,154 +0,0 @@
-# frank-archon — Radiance (StillDeadcode), publisher Qwen3.8-Flash-Next iq4r, 2× R9700 TP=2
-
-**Status: LIVE — deployed 2026-10-15.** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 2× Radeon AI PRO R9700 (gfx1201), TP=2
-
-The in-house w4dn8 conversion of the Swift1.5 finetune
-([[frank-archon-radiance-swift15-w4dn8]]) is replaced by the **engine
-author's published container**:
-[`StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe`](https://huggingface.co/StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe)
-— the official `Qwen/Qwen3.8-Flash-Next` bf16 checkpoint rad-converted by
-StillDeadcode. Same port, alias, container name, systemd unit, digest-pinned
-engine pin **1.0.13** (`stilldeadcode/radiance@sha256:4c3f484564…`), and
-**every serving argument carried verbatim** from the 2026-10-13 shape.
-Rollback: re-download the w4dn8 `.rad` from the HF mirror + point
-`MODEL_FILE` at it (the local copy was purged 2026-10-17, see Rollback).
-
-## The payload
-
-`qwen3.8-next-flash-fp8-iq4r-moe.rad` — **121,969,901,568 B = 116,322 MiB**,
-sha256 `0af5e96244e80c21ac8edca1719dae49b8a201b94a2f3994c3e24026ceaa4d20`
-(Hub LFS oid), fetched byte-exact by `scripts/download-frank-archon-iq4r.sh`
-(hfget resume + size + sha256; staging dir
-`/home/dank/models/StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe/` keeps the
-card, LICENSE, and recipe `qwen4exp-w4nl64-i8-hc8m` as provenance).
-
-| Plane | Encoding (publisher recipe `qwen4exp-w4nl64-i8-hc8m`) |
-| --- | --- |
-| Routed experts | 4-bit codes into a 16-level **non-uniform** codebook after a 128-point Walsh–Hadamard rotation, one E4M3 scale per 64 weights; codes by GPTQ (10M-token calibration, per-expert down-proj Hessians) + 3 coordinate-descent sweeps |
-| Protected experts | **ten** per layer, whole-expert bf16 (down-projection energy) |
-| Trunk + lm_head | int8 W8A8, scale per 128 columns searched; HC mixtures E4M3 |
-| PLE n-gram | fp8 fixed-scale, in-container (served `--ngram-placement ram`, same 47.7 GiB pin law) |
-| Speculator | the model's own MTP head (card default depth 3; served at 2) |
-| Vision | bf16 vision tower, in-container |
-
-**Base model: the OFFICIAL (censored) Qwen3.8-Flash-Next** — the fleet's
-only frontier model at base alignment; the abliterated family lives on
-frank-dragoon. Card quality: full-vocab teacher-forced KL vs bf16 over
-~75K positions — mean 0.0789 all-tokens / 0.0239 assistant turns, top-1
-agreement 92.85 %.
-
-## Why this swap
-
-- **Publisher-graded precision**: the w4nl codebook + coordinate-descent
-  refinement is the engine author's own quality ceiling for this
-  checkpoint; the card's KL ladder (p99 1.65, p99.9 5.33) is published and
-  reproducible, where our w4dn8 gate measured against our own reference.
-- **Formal parity**: the served file is the one the engine author supports
-  — decode/prefill behavior on 1.0.13 is their tested path (the card's run
-  line speaks the 1.0.13-era flags; the `--vram-weights-mib`/`--vram-kv-mib`
-  pair we ship is accepted by pin 1.0.13 per the 2026-10-06 A/B, so the
-  shape needed no translation).
-- **Cost**: the Swift1.5 finetune's refusal-bending behavior leaves this
-  endpoint (it returns to the official base alignment); the 4-slot/200k
-  economics and all shape laws carry over unchanged.
-
-## Shape (unchanged from 2026-10-13)
-
-All of it, verbatim — the payload moved 116,416 → 116,322 MiB, inside the
-same budget:
-
-- `--tp 2 --tp-wire wht6`, `--max-num-seqs 3`, `--max-model-len 200000`
-  (capacity contract 3 × 200,000), `--kv-cache-dtype fp8`
-- `--placement expert_tiered --host-pool-mib 44928` + explicit
-  `--vram-weights-mib 25600 --vram-kv-mib 5120` (30,720 of the engine's
-  30,969 MiB ceiling; pool = contract + 1.3 %)
-- `--num-speculative-tokens 2`, `--max-num-batched-tokens 2048`
-  (MNBT 2048 law), prefix cache 8 GiB host + 64 GiB disk,
-  `--ngram-placement ram`, froggeric v22.5 `--override-chat-template`
-- TTM preflight floor 150,000 / target 204,800 MiB; pinned ledger
-  2 × 44 GiB pool + 47.7 GiB PLE + 8 GiB prefix ≈ 147 GiB (see
-  [[frank-archon-radiance-swift15-w4dn8]] for the ledger arithmetic and
-  the KV dtype law — both are engine properties, payload-independent)
-
-The publisher's card run line (`--host-pool-mib 12288`,
-`--expert-vs-cache-ratio 0.82`, seqs 8, spec 3, ~32 GB host RAM) is their
-shape on stock 32 GB cards; the fleet ships its own measured shape against
-the same file.
-
-## Cutover record — 2026-10-15
-
-1. `rm` (Frank's one-time authorization): the raw
-   `ukisai/Swift1.5-Qwen3.8-Flash-Next` BF16 checkpoint (336 GiB, spent
-   conversion source) and `qwen4exp-fp8-native.rad` (173 GiB, two eras
-   back) left `/home/dank/models` — 150 GB free → 658 GB free. The fp8-native
-   payload is **no longer on disk**: rebuild =
-   `scripts/download-frank-archon.sh` + `scripts/build-frank-archon-rad.sh`.
-2. Fetch: `scripts/download-frank-archon-iq4r.sh --fetch` as transient unit
-   `iq4r-fetch` (detached-job protocol; log
-   `/home/dank/logs/iq4r-fetch.log`, rc sentinel `…rc`).
-3. Deploy: `scripts/deploy.sh dankserver --service frank-archon` — launched
-   detached (the swap restarts the very endpoint the operator session runs
-   on; the harness timeout catches the session on return).
-4. **Boot ledger (2026-10-15, first boot, pin 1.0.13 — the w4nl codebook
-   loaded, no pin bump)**: `kv_attn` fp8 page **26.00 KiB** at **607,740
-   tokens** (151,935 blocks = 3.77 GiB) — byte-identical to the w4dn8 boot,
-   the sizing law carries; `kv_qsa_bkey` 3.25 KiB at the same token count;
-   `kv_gdn_state` 54 MiB × 6 slots, `kv_gdn_conv` 1.76 MiB × 3,
-   `kv_qsa_tail` 19.5 KiB × 6; pool **5.00 GiB**, prefix checkpoints 8 ×
-   56 MiB; the sizing line reads `kv cache 5.00 GiB … backs 3 sequences all
-   at 200000 tokens at once, which is the worst case`; samplers print **9
-   rows** per rank (3 × (1+2), inside the wht6 25-row bound);
-   `speculation: 'mtp' drafts 2 token(s) a step in 2 serial pass(es)` +
-   prompt-lookup fallback; slab **17,610 (rank0) / 17,658 (rank1) slots**
-   (64 shadow) — above the w4dn8 record of 17,548 (ten bf16-protected
-   experts per layer replace twenty-four, so more of the 4-bit table
-   resides), pinned pool 7,532/7,484 slots; KV lending **4.22 GiB**
-   (3,571/3,568 mover slots); arenas 896.02/988.17 MiB charged before the
-   pair split; prefix cache RECOVERED 900,176 blocks + 126 checkpoint slots
-   from the 64 GiB disk tier (same arch + tokenizer as w4dn8 — the tier
-   survived the payload swap); froggeric template 28,234 B; vision encoder
-   live (≤ 4,194,304 px, 16,384 patches/pass). Health gate passed in ~100 s
-   (radiance binds the port after the full load).
-5. **Vision probe (2026-10-15)**: 300 px solid-red PNG via `image_url` —
-   reasoning stream: "a solid fill of a deep, saturated red hue". Content
-   came back EMPTY at `max_tokens 64`: the thinking budget consumed all 64
-   (`reasoning_tokens: 64`, finish `length`) — fleet law, probe thinking
-   models with ≥ a few hundred tokens.
-6. **e2e gate**: deploy exited DEPLOY INCOMPLETE — 30/33 Playwright green,
-   3 red (`12-crawl`, `13-sso` search SSO, `16-marreta`) on DANKCLAW
-   surfaces (vhosts intermittently returned `ERR_CONNECTION_RESET` from the
-   workstation); frank-archon's own health gate passed. The swap touched no
-   dankclaw runtime file (only homepage services.json text). Census of the
-   Caddy/auth stack pending; see `log.md`.
-7. **Post-boot measurements (pending)**: decode ladder (99.0 t/s single
-   @ 43.5k, 93.3 @ 197k, 3-way 113.9/43.3/10.9 = 18.4 t/s aggregate —
-   these are the **w4dn8** numbers, the record to beat, not this file's),
-   slab-limited prefill (22208/3,397 → 25600/3,584 t/s A-cold 43.5k), draft
-   acceptance, and `scripts/probe-frank-archon-kv.py` leg D.
-
-## Rollback
-
-Revert the swap commit (`scripts/frank-archon.env` MODEL_FILE line) +
-redeploy: `/home/dank/models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad`
-(116,416 MiB), KLD-gated, same shape — **no longer on disk**. The 2026-10-17
-disk purge (Frank's approval, trash-staged then emptied: 763 GiB free after)
-removed it; the byte-exact copy lives on the Hub as
-`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`
-(`qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`)
-and re-fetches through `scripts/download-frank-archon-iq4r.sh`'s
-hfget/verify pattern retargeted at that repo. The same purge deleted the
-`radiance-state` kvcache tiers, `~/.radiance-cache-w4a8-*`,
-`~/.cache/frank-primary`, and `/opt/vllm/moremodels` (GLM-4.7 GGUF) — all
-derived or retired. The fp8-native era is rebuild-only (see the cutover
-record). The w4dn8 runbook keeps every shape law, the KV dtype table, the
-slab ladder, and the tuning levers:
-[[frank-archon-radiance-swift15-w4dn8]].
-
-## Related
-
-- [[frank-archon-radiance-swift15-w4dn8]] — the replaced payload and the
-  full shape-law library (slab slope, KV dtype arithmetic, TTM law)
-- [[frank-archon-radiance-qwen38-native-fp8]] — the fp8-native era
-  (rebuild-only now; owns the descriptor-selection law)
-- [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here
-- [[deployment]] — the service-scoped deploy this cutover rides

serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-native-fp8.md

@@ -1,308 +0,0 @@
-# frank-archon — Radiance (StillDeadcode), Qwen3.8-Flash-Next-FP8 NATIVE, 2× R9700 TP=2
-
-**Status: RETIRED 2026-10-05 → 2026-10-08** — replaced by the Swift1.5 w4dn8
-conversion ([[frank-archon-radiance-swift15-w4dn8]]), itself retired 2026-10-15;
-the payload live today (2026-10-18) is the publisher iq4r container,
-[[frank-archon-radiance-qwen38-iq4r]] (`scripts/frank-archon.env:110-111`). ·
-**Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 2× Radeon AI PRO R9700 (gfx1201), TP=2
-
-The FreeToken PR #587 endpoint was replaced by the **radiance engine** (C++/HIP,
-libr4d kernels for gfx1201, no Python in the serving path) driving the
-digest-pinned image `stilldeadcode/radiance@sha256:4c3f484564bb2872784e4…`
-(v1.0.7 line, 287 MiB, `libr4d@0.5.0`) as a plain `podman run` through
-`scripts/serve-frank-archon.sh` under the **root systemd unit**
-(`frank-archon.service`, `User=dank`, `LimitMEMLOCK=infinity`). Same port,
-same served alias (`frank-archon`), same container identity
-(`frank-archon` = the unit name). Rollback: git tag
-`frank-archon-freetoken-pr587-20261005` (FreeToken trio + Dockerfile; the
-Dockerfile lives in the tag as `scripts/docker/frank-primary-freetoken.Dockerfile`
-— `scripts/docker/` is not in the tree today), image
-`localhost/frank-primary-freetoken:pr587-77a7201` (that is the tag on the host —
-the 2026-10-10 rename never retagged it), payload
-`/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8`, which is **no longer on
-disk** (`ls /home/dank/models/` on Dankserver 2026-10-18: `StillDeadcode`, `rad`,
-`tpl.jinja` only).
-
-## Why radiance
-
-- **TP=2 is real here.** FreeToken's TP=2 dies in the `pynccl.cu` ROCm JIT
-  (`cuda_runtime.h` unresolvable for gfx1201 — probed 2026-10-06); the
-  second R9700 idled. Radiance's TP wire is libr4d's own one-shot
-  all-reduce (no RCCL), legal on this board since the 2026-09-20 swap
-  (ProArt B850, x8/x8 on CPU root ports, `Routing+`/`ReqEn+` everywhere).
-- **Native FP8 end to end** (Frank's call): the official
-  `Qwen/Qwen3.8-Flash-Next-FP8` checkpoint keeps its modelopt block-FP8
-  encoding — no requantisation, no calibration run, no 4-bit codebook.
-- **Every tier is explicit and measured**, not an estimator: placement
-  `expert_tiered` with a pinned host pool, `--ngram-placement ram` for the
-  47.7 GiB PLE table, `--expert-vs-cache-ratio 0.82` (the publisher's
-  measured 99 %-hit point; 0.87 already collapses to 0 %), fp8 KV.
-
-## The payload: native FP8, reshaped then converted (one recipe rule)
-
-- **Raw checkpoint**: `/home/dank/models/Qwen/Qwen3.8-Flash-Next-FP8`
-  (ungated repo, 172.8 GB, 131 shards + index + tokenizer + vision
-  configs). Fetched by `scripts/download-frank-archon.sh` (vendored
-  `hfget.sh`, curl+jq, no HF token needed) and **verified byte-for-byte**:
-  all 131 shards sha256-match the Hub LFS oids (2026-10-05, log
-  `/home/dank/logs/verify-qwen38-official-fp8.log`).
-- **The container**: `/home/dank/models/rad/qwen4exp-fp8-native.rad`
-  (177,109 MiB measured = 173.0 GiB; floor
-  `FRANK_ARCHON_RAD_MIN_MIB=177000` in the env gates a
-  truncated convert). Built by `scripts/build-frank-archon-rad.sh`:
-  1. stage `/home/dank/models/stage-qwen4exp-fp8` — `config.json` with
-     **`quantization_config` dropped** + relative symlinks into the raw
-     checkout;
-  2. `stack-qwen38-official-fp8.py` reshapes the staging dir — two
-     transforms, both value-preserving (script docstring carries the full
-     argument): the plugin maps the STACKED expert layout
-     (`mlp.experts.gate_up_proj` `[512,1280,2560]` + `_scale_inv` at the
-     same index) while the official repo ships per-expert unfused tensors,
-     so staging byte-stacks all 49 stacks' (48 trunk + MTP) 512 experts
-     with NO requantisation; and the PLE table ships as 128 **scaleless**
-     E4M3 shards + one bf16 scalar (1.9931793212890625e-04) while the
-     plugin's fp8 branch wants the codes beside a ONE-ELEMENT bf16 scale
-     plane — so staging decodes to a single bf16 `shard_0` with the
-     checkpoint's own scale (e4m3 decode and the scale are both exact in
-     bf16);
-  3. `rad-convert` with ONE `--quant` rule — pure copy for everything else
-     (every weight keeps the checkpoint's block-FP8 planes: `X.weight`
-     E4M3 codes + `X.weight_scale_inv` bf16 `[128×128]` scales) — and the
-     rtn rule re-encodes the bf16 PLE table at the same fixed scale, an
-     exact round-trip (nearest-code distance 2⁻³ vs the bf16 rounding
-     perturbation 2⁻⁹): the container gets the checkpoint's own PLE codes
-     plus the one-element scale plane (`blk.1.ple_ngram.scale`, view {1});
-  4. THREE LAWS the live runs proved (each cost a full verify+declare
-     cycle; all fixed in the two scripts, 2026-10-05/06):
-     - **Scale-name law**: `scale_name()` appends `_scale_inv` to the
-       TENSOR NAME verbatim — a stacked tensor named
-       `…experts.gate_up_proj` (no `.weight` suffix) has its scale plane
-       at `…experts.gate_up_proj_scale_inv`, NOT `…weight_scale_inv`.
-     - **File-shadow law**: the reader opens every index-named file in
-       SORTED order and emplaces EVERY tensor each file physically
-       holds, first-wins (`checkpoint.cpp add_shard`). The raw
-       model-000NN shards carry tensors NAMED exactly what the merged
-       PLE shard is named, so the merged file MUST sort first
-       (`00-ple.merged.safetensors`) or a raw fp8 shard wins the name
-       and the plan reads one shard as the codes.
-     - **Symlink-through law**: staging symlinks the raw index into the
-       stage; `open(path,"w")` writes THROUGH it and corrupts the raw
-       checkpoint (raw verify then fails 143/144). The stack script
-       writes index/config temp+`os.replace` — an atomic swap replaces
-       the symlink itself; the build script COPIES the index and
-       symlinks only the shards.
-  5. `rad-info` summary + size floor.
-- **Why the staging dir is mandatory** (the descriptor law, spec §2.4):
-  the plugin is selected by the exact `(arch_id, quant descriptor)` pair
-  with no fallback. A raw official checkpoint carries
-  `quantization_config.quant_method: "fp8"` → descriptor `"fp8"` → **no
-  shipped plugin claims `(qwen4exp, "fp8")`** — raw serve AND unmodified
-  convert both refuse. The shipped `qwen4exp_fp8` plugin claims
-  `(qwen4exp, "")` (recipe-built containers). `rad_convert.cpp` writes
-  `w.set_quant(meta.quant)` into the header, so the staged config's empty
-  quant method flows through as the container descriptor while **every
-  weight stays FP8 data**. Native precision is untouched; only the
-  selection key differs.
-- `--plan-only` first: the convert's declare phase is the engine's own,
-  so a plan that resolves every op will boot. The convert runs card-less
-  (declare = CPU-side layout queries); add `--device` only if a plugin
-  load ever demands it.
-
-## The shape (`scripts/frank-archon.env`, every value commented there)
-
-| Knob | Value | Why |
-|---|---|---|
-| Image | `stilldeadcode/radiance@sha256:4c3f4845…` | digest-pinned INDEX digest (RepoDigests also carry the per-arch manifest `1c582005…`; the image ID `69d05724…` is the config digest, NOT a manifest digest — pinning it sends `podman run` to the network); entrypoint = engine, tools (`rad-convert`/`rad-info`/`rad-tune`/`rad-kbench`) on PATH |
-| Container | `frank-archon` | identity = service name across runtimes; legacy list retires `frank-archon-qwen frank-archon-r9v qwen-fp8 glm-probe` |
-| `--tp 2 --tp-wire wht6` | both R9700s | fused WHT-6 lossy wire: +5.6–6.3 % prefill, ~−1.3 % 8-stream decode at TP2; `--tp-wire-min-kb` stays engine default (128 KiB) |
-| `--max-num-seqs 2` | first boot | Frank: "parallelism = 2 to start". Raise only after a real-traffic look at slot occupancy + the ledger's KV headroom line. The fused all-reduce row count is `seqs × (1 + spec) = 8` |
-| `--max-model-len 200000` | served window | native 262,144; the publisher's production value and what `models.yml` quotes. Longer prompt = refused, never truncated |
-| `--placement expert_tiered --host-pool-mib 40960` | pinned expert pool **per rank** | native FP8 experts are 112.8 GiB total (56.4 GiB/rank slice) vs ~29 GiB at the 4-bit publisher shape — the pool is the price of native precision. Resize from the boot ledger's coverage line, never arithmetic |
-| `--gpu-headroom-mib 96 --expert-vs-cache-ratio 0.82` | publisher values | 0.82 = measured 99 % expert-cache hit; 0.87 → 0 % (the cliff) |
-| `--kv-cache-dtype fp8` | **7.32 KiB/token** | 12 QSA full-attention layers, 2 KV heads × 256 dim, K+V; a 200K-token sequence is 1.43 GiB. KV is sized from the VRAM the ratio leaves |
-| `--num-speculative-tokens 3` | MTP head | the checkpoint carries `mtp.*` (3,101 tensors, `mtp_num_hidden_layers: 1`); the plugin's probe returns draft depth 3 — the publisher's operating point |
-| `--max-num-batched-tokens 2048` | prefill chunk | A/B-verified optimum (1024/2048/4096 ladder, see "Tuning levers") |
-| `--prefix-cache-host-mib 4096 --prefix-cache-dir /data/kvcache/frank-archon --prefix-cache-disk-mib 65536` | 3 tiers | host tier pinned (see the ledger), disk tier in `/home/dank/radiance-state` (mounted rw at `/data`; `/` is ext4, mapped-tier-eligible); a session snapshot moves with its blocks |
-| `--ngram-placement ram` | **PLE pinned to host RAM** | Frank's pin law. The PLE n-gram table is 51.2 GB (320,001,536 rows × 160 dims = 29 % of all parameters) served via io_uring reads; `ram` **refuses the boot** if memlock is too small |
-| `--override-chat-template /templates/template.jinja` | froggeric v22.5 | vendored `scripts/qwen38-froggeric-v22.5.jinja` (28,234 B, `qwen3.8-froggeric-v22.5`); rad-convert bakes the checkpoint's own template, this overrides it, and the engine derives the reasoning/tool-call reply format from it |
-
-**Pinned-RAM ledger** (root unit + `LimitMEMLOCK=infinity` exist for this):
-2 × 40 GiB expert pools + 47.7 GiB PLE + 4 GiB prefix host tier + ~2 GiB
-spare ≈ **134 GiB**. `ttm.pages_limit` (this kernel has NO `pages_max` —
-the two-knob story was stale) caps GPU-mappable pinned pages (default = half
-of RAM): the pre-cutover cap was **123 GiB** — BELOW the ledger — so the
-serve script's TTM preflight (`FRANK_ARCHON_TTM_MIN_MIB=140000`) refuses to
-start until it is raised. DONE 2026-10-05 pre-cutover: sysfs write of
-**52,428,800 pages = 200 GiB** (`/sys/module/ttm/parameters/pages_limit`;
-verified read-back; raising at runtime is harmless and allocates nothing).
-PERSISTENCE: the sysfs write does not survive reboot, and the kernel-cmdline
-`ttm.pages_limit=52428800` was never added — the unit now raises the knob at
-every boot instead (`scripts/frank-archon.service:44`,
-`ExecStartPre=+…/scripts/ttm-preflight.sh --raise`).
-
-**Per-rank VRAM** at TP2: expert slice 56.4 GiB total across VRAM + host
-pool; trunk ~3.75 GiB/rank; token embedding vocab-parallel (~1.18 GiB/rank
-— the bf16 table is 2.43 GiB, sharding hands the other half to KV); lm_head
-row-sharded; recurrent state 54 MiB/copy/card (2 copies with speculation);
-KV from the `0.82`-side remainder of claimed VRAM at 96 MiB headroom.
-
-## The engine contract (all verified in source, not assumed)
-
-- **`/health` == `/ping`**: 200 JSON (`model_id, in_flight, queue_depth`)
-  while serving, 503 only while stopping — **the listener binds AFTER the
-  full load**, so a bare `curl -fsS http://127.0.0.1:7770/health` gate is
-  honest (the deployer loops on it; the FreeToken-era `"status":"ok"` body
-  grep is gone). Boot budget 1800 s (publisher `start_period` 15 m);
-  measured 2026-10-06: health 200 ~2-3 min after start (~120 s warm) —
-  the big terms are the 125.26 GiB page-in (38.1 s at 3.5 GB/s) and the
-  host-pool pin (PLE 47.68 GiB in 43.9 s).
-- **Sampling** (oai.cpp): `temperature, top_p, top_k, min_p, typical_p,
-  presence_penalty, frequency_penalty, repetition_penalty, repeat_last_n,
-  chat_template_kwargs, reasoning_effort, enable_thinking,
-  preserve_thinking` accepted; **unknown params are refused (400)** —
-  models.yml's extraBody set is fully inside it; nothing unverified rides
-  the wire. `max_tokens` is clamped, not refused; `seed` absent = pick
-  one; `response_format` supported; **`logprobs` is refused** (harness
-  never sends it).
-- **Reasoning**: `--reasoning-format` defaults to `auto` → thinking in
-  `reasoning_content` (models.yml `reasoningContentField` matches). Tool
-  calls: the qwen3.8 dialect → `message.tool_calls` +
-  `finish_reason: tool_calls`. No parser flags anywhere — parsers are
-  auto.
-- **Vision**: the official checkpoint carries the 27-layer vision tower;
-  the engine runs the Qwen-VL processor (patching, pad-token expansion,
-  3-part rotary are core). `models.yml` keeps `input: [text, image]`;
-  `--mm-max-patches` available.
-- **`/metrics`**: vLLM-compatible names + `radiance:*` families — the
-  Dankclaw Prometheus job re-scrapes 192.168.0.235:7770 (plain LAN routing
-  reaches it; the netavark published-port quirk is same-host only).
-  `/stats`, `/sessions`, `/dashboard` for live internals; the repo ships
-  `deploy/grafana/radiance.json` if a dashboard is wanted.
-- **`n` > 1 requests**: `n` defaults to `max(8, max-num-seqs)`; stop
-  strings ≤ 64 × 4096 B. `model` field not enforced.
-
-## Deploy + verify
-
-```bash
-# one-time (already done 2026-10-05): payload fetch + sha256 verify + convert
-ssh dankserver 'bash /home/dank/frank-archon-build/build-frank-archon-rad.sh --plan-only'   # declare + plan, no bytes
-ssh dankserver 'bash /home/dank/frank-archon-build/build-frank-archon-rad.sh'               # full convert + rad-info
-./scripts/deploy.sh dankserver --service frank-archon
-```
-
-Post-cutover checklist (each is observable, not assumed):
-
-1. Unit: `is-active` active, `is-enabled` enabled, container answers
-   `/health` with 200 JSON.
-2. Boot ledger (engine stdout → `/home/dank/logs/frank_archon.log`):
-   P2P probe per rank pair, TP=2 shard placement, expert pool coverage
-   line ("backs N % of the worst case … M full-length sessions"), KV pool
-   bytes, PLE pinned-RAM line, drafter depth 3, prefix tiers.
-3. MTP live: a completion's `timings` object shows draft/accept counts;
-   `radiance:*spec*` metrics in `/metrics`.
-4. Froggeric template: a probe prompt renders the v22.5 shape
-   (tool-call + thinking turn both exercise it).
-5. Vision: a ≥256 px solid-color PNG answers the exact color through an
-   `image_url` content part.
-6. Prometheus target `frank-archon` green (not never-green).
-
-## Cutover measured (2026-10-06, all values observed live)
-
-- Container: 1,857,127,997,440 B = 177,109 MiB = 173.0 GiB. Plan-only
-  priced 172.76 GiB — the DATA planes; the written container adds a
-  metadata tail. Floor gates must be set from the real file (the first
-  deploy refused on 178000 > 177109; floor is 177000).
-- Boot: 125.26 GiB read in 38.1 s (3.5 GB/s), PLE pinned
-  "47.68 GiB in 43.9 s" (the 2 B scale plane with it), each rank's
-  mover holds 25.46 GiB static+experts (19.15 GiB resident experts,
-  79 %), 16,820 pinned pool slots, health 200 ~2-3 min after start
-  (~120 s warm). No 7-minute vLLM-style boot.
-- KV ledger: kv_attn 99,956 blocks = 2.48 GiB (399,824 tokens),
-  kv_qsa_bkey 317.24 MiB, kv_gdn_state 4 slots × 54 MiB, PLE conv
-  states < 1 MiB; linear checkpoints 56.37 MiB × 8 slots (3 of 8
-  card-backed at boot, 5 withdrawn under the 208 MiB floor).
-- MTP on real traffic (temp 0.7, 220-token ask): draft_n 225 /
-  draft_n_accepted 119 = 52.9 % acceptance, decode 23.8 t/s. ONE
-  sample — the shape's real number needs live-traffic measurement.
-- Contracts: `reasoning_content` field (matches models.yml), usage
-  metering separates reasoning_tokens; vision — a 512² solid-red PNG
-  through `image_url` answers `Red` (279 prompt tokens; NOTE the
-  32-token budget is eaten by thinking — give vision probes ≥256
-  max_tokens); native `tool_calls` with finish_reason tool_calls.
-- Boot defects fixed the same day: the SSD prefix tier opened only
-  after the serve script mkdir -p'd the full path (the engine's
-  create is NOT recursive: "cannot create /data/kvcache/..."); the
-  converter's scale-name law, the PLE file-shadow and the
-  symlink-through-raw-index bug are in the payload section above.
-- Prometheus: `/metrics` answers 200 with vLLM-compatible families
-  (verified from Dankclaw); the `frank-archon` job needs a
-  prometheus-service deploy to go green (config pushed, reload
-  deferred behind a live parallel writer's dirty tree).
-
-## Tuning levers (in order of likely use)
-
-- **`FRANK_ARCHON_MAX_SEQS`**: raise 2 → 3/4 only when the ledger's KV
-  headroom line and real slot occupancy agree; each slot buys a
-  recurrent-state copy (54 MiB/card) and shrinks the per-request KV share.
-- **`FRANK_ARCHON_HOST_POOL_MIB`**: the ledger's coverage line is the
-  oracle — it prints what the pool backs at the worst case and how many
-  full-length sessions fit. RAM budget: pools + PLE + prefix ≤ ~140 GiB of
-  the host's 246.
-- **`FRANK_ARCHON_EXPERT_VS_CACHE`**: 0.82 is the measured 99 %-hit
-  point; 0.87 zeroes it. Move only with a measured hit-rate reason.
-- **`FRANK_ARCHON_TP_WIRE`**: `wht6` vs `exact` — `exact` trades the
-  +5.6 % prefill for bit-exact reductions; the fused one-shot forms are
-  declared over the seqs × (1+spec) row bound.
-- **`FRANK_ARCHON_EXTRA_ARGS`**: the boot-ladder escape hatch, expanded
-  unquoted.
-- **`rad-tune -m <rad> --tp 2 --num-speculative-tokens 3`**: benchmarks
-  the tunable space at this model's shapes on this machine, writes
-  `$RADIANCE_HOME/tune/<machine>.tune` which the engine's selector reads at
-  boot. Declare at the shapes you serve (`--tp 2
-  --num-speculative-tokens 3`), or you measure a program the serve never
-  runs. `$RADIANCE_HOME` is `/opt/radiance/share/radiance` — in the IMAGE — so a
-  `--rm` scratch container's cache dies with it: the serve script
-  bind-mounts `$FRANK_ARCHON_STATE_DIR/tune` (`/home/dank/radiance-state/tune`)
-  at `/opt/radiance/share/radiance/tune`, so the cache (36 GEMM rows,
-  `gfx1201-x3-AMD_Radeon_Graphics.tune`, generated 2026-10-06 in a
-  stopped-service window) survives and is read at every production boot.
-  The two attention requests have no shape recipe (`attn_paged`,
-  `attn_paged_gate_quant` — "see tools/opshapes.h") and are not tuned.
-  `RADIANCE_TUNE` pins a kernel for an A/B — an instrument, not a setting.
-- **`FRANK_ARCHON_MAX_BATCHED_TOKENS` (MNBT)**: A/B'd 2026-10-06 in a
-  stopped-service window (real-text probes: 20,417-token cold prefill,
-  1,985-token warm prefill, 256-token decode). The prefill ladder is
-  NON-MONOTONIC with an interior optimum at 2048: 1024 → 2048 → 4096 =
-  231 → 460 → 448 t/s cold, 221 → 412 → 427 t/s warm. Decode is flat within
-  noise (24.0 / 25.1 / 24.3 t/s); draft acceptance 60.1 / 67.1 / 63.3 %.
-  **2048 wins** (peak prefill + peak acceptance) — it stays the production
-  value. 1024 halves prefill (the batch too small to saturate the card);
-  4096 is a hair off 2048 (the chunk past the kernel's sweet spot).
-- **Diagnostics**: `rad-info` (container: encodings, recipe, profile),
-  `--debug-graph` (what declared vs what issued), `rad-kbench` (kernel
-  correctness/speed vs libref), `/stats` + `/sessions` (live), and the
-  publisher's `scripts/` harnesses (`fnpf`, `fnconc`, `cstep2`, `ident`,
-  `mctools`) — note `mcserve/fnstop`-family scripts `pkill` the engine
-  binary by name; on this host the service is a systemd unit, so use
-  `sudo systemctl stop frank-archon` instead.
-
-## Rollback
-
-| Layer | Path |
-|---|---|
-| FreeToken (one card, native FP8 safetensors) | `git checkout frank-archon-freetoken-pr587-20261005 -- scripts/frank-archon.env scripts/frank-archon.service scripts/serve-frank-archon.sh scripts/docker/ tests/frank_archon_unit.bats` + redeploy; the image in storage is `localhost/frank-primary-freetoken:pr587-77a7201`, and the payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` must be re-fetched — it is gone from the host |
-| tcclaviger vLLM fork (TP=2, shape G) | git tag `frank-archon-vllm-shapeG-20261003` (trio + env lore; image still in dank's rootless storage) |
-| 4-bit StillDeadcode publisher shape | `qwen3.8-next-flash-fp8-iq4r-moe.rad` — **this became the live payload on 2026-10-15** ([[frank-archon-radiance-qwen38-iq4r]]), deployed with `--host-pool-mib 44928` (`scripts/frank-archon.env:184`), not the 12,288 MiB first sketched here |
-
-The FreeToken-era orcarouter payload has since been deleted from the host
-(`ls /home/dank/models/` on Dankserver, 2026-10-18) — there is nothing left to
-trash, and the FreeToken rollback needs a re-fetch.
-
-## Links
-
-- [[LLM/index]] — fleet index
-- [[frank-archon-freetoken-pr587]] — the retired FreeToken era
-- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the retired tcclaviger vLLM era
-- [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here
-- [[RDNA4 Container Serving]] — rootless-podman GPU laws on these hosts
-- `serverAppConfigs-docs/Architecture/Testing Policy.md` — the seam contract

serverAppConfigs-docs/LLM/frank-archon-radiance-swift15-w4dn8.md

@@ -1,625 +0,0 @@
-# frank-archon — Radiance (StillDeadcode), Swift1.5 Qwen3.8-Flash-Next W4Dn8, 2× R9700 TP=2
-
-**Status: RETIRED — superseded 2026-10-15 by the publisher iq4r payload.** · Lived 2026-10-08 → 10-15 (main `3ca64e1`, e2e 33/33). · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 2× Radeon AI PRO R9700 (gfx1201), TP=2
-
-The FP8-native era (`qwen4exp-fp8-native.rad`, 173.0 GiB, no-recipe convert)
-is replaced by a **GPTQ-calibrated 4-bit container** of
-`ukisai/Swift1.5-Qwen3.8-Flash-Next` (BF16, 335.3 GiB, abliterated
-finetune) converted with the recipe
-`scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe` on engine
-**1.0.13** (`stilldeadcode/radiance@sha256:4c3f484564…` — same digest as
-the FP8-native era; 1.2.0 was used ONLY for the calibration serve).
-Same port, alias, container name, systemd unit, and every serving argument
-carried into the successor: [[frank-archon-radiance-qwen38-iq4r]]. The
-`.rad` itself was purged from disk 2026-10-17; the byte-exact payload
-lives on the Hub — see Rollback.
-
-## Why this swap
-
-- **Context economics at production memory budgets**: the FP8-native
-  experts (56.4 GiB/rank slice) forced a 40 GiB/rank pinned host pool and
-  left KV ~3.5 GiB. W4Dn8 experts cut the resident set to ~1/4, freeing
-  host and VRAM budget for KV and slots at equal-or-better measured
-  decode (gate: ≥ 30.8 t/s vs the 30.08 t/s control).
-- **Swift 1.5 base**: abliterated finetune of Flash-Next (Frank's
-  fleet-personality call; frank-dragoon serves the same family).
-- **GPTQ closes the precision gap**: RTN alone on BF16 weights lands
-  measurably worse; GPTQ with Hessians from a 10M-token corpus of this
-  model's own distribution recovers most of it, verified end-to-end by a
-  KLD gate against the BF16 container itself (KL ≤ 0.03, top-1 ≥ 95 %).
-
-## The payload chain
-
-1. **Fetch**: `scripts/download-frank-archon-swift15.sh` →
-   `/home/dank/models/Swift1.5-Qwen3.8-Flash-Next` (131 shards, 335.3
-   GiB, per-shard LFS sha256 verified, no token on the command line).
-   48 layers, 512 experts (top-10 routed), moe_inter 640, hidden 2560,
-   PLE 20M-vocab table 95.37 GiB, MTP head, vision tower.
-2. **BF16 container**: `run-swift15-bf16.sh convert` →
-   `rad-convert` no-recipe (pure copy) at `--max-ctx 4096` →
-   `qwen4exp-swift15-bf16.rad` (335.28 GiB). Serves on 1.0.13 as the
-   KLD reference engine (disk-tier placement; boot ~35 min).
-3. **KLD reference**: `run-swift15-bf16.sh kld` (1.0.13) →
-   `/home/dank/corpora/kld-ref` — `radiance-kld-1` format, `rows:
-   50235`, 197 source-labelled docs, top-16 log-probs (f16), corpus
-   fnv1a64 `a9d62b33c18b3d9c`. ~4 h at 32k prefill steps
-   (`--max-num-batched-tokens 32768` — a 65536 step OOMs an arena
-   tensor at `--max-model-len 16384`). **The wrapper's exit line is the
-   receipt: never scp this script while it runs** (bash lazy-read;
-   a mid-run swap killed a clean engine exit with rc=127, third fleet
-   occurrence of the script-swap law). `kld-then-calib.sh SKIP_KLD=1`
-   resumes from the `rows: 50235` receipt.
-4. **Calibration**: `run-swift15-bf16.sh calib` on **1.2.0** (the only
-   engine with `RADIANCE_CALIB_TOKENS`; grams are engine-invariant
-   bf16 activations — RADGRAM2 format byte-identical across versions)
-   + `calib-push.py` pushing `calib-corpus.jsonl` (10,014,932 tokens,
-   1,294 prompts, assembled from real HF sources, chat-templated,
-   source-labelled). The tap (in the `qwen4exp_fp8.so` architecture
-   plugin) writes per-expert down-proj Gram matrices
-   `gram.r<rank>.blk.L.ffn_down_exps.<e>.bin` (32-byte RADGRAM2 header +
-   640×640 f32; `rows` = routing count, `trace` = activation energy)
-   and stops the serve at the token threshold. TP=2: whole experts per
-   rank (256/rank); gate_up input is rank-sliced (plugin warning — the
-   recipe only casts the down projection).
-5. **Protected-expert derivation**: `derive-protected-experts.py
-   /home/dank/corpora/calib` → the 24 highest-energy experts (the law
-   is data-driven; Swift drew 24, not the fp8-era 10 — its routing is
-   spikier: `blk.47 e445` alone carries 32 % of its layer's down energy
-   at 70× routing share) promoted to whole-expert bf16.
-   Protection law: energy share ≥ 3 % AND ≥ 3× routing share; even count
-   per rank half; odd half gains the next most energetic.
-   `apply-protected-cast.sh` splices the derived names between the
-   `# PROTECTED-BEGIN/END` markers of the recipe (refuses double-derive).
-6. **Final convert**: `convert-swift15-w4dn8.sh` (1.0.13, host-backend
-   CPU-only — no GPU visible to `rad-convert`; GPTQ solve is CPU BLAS) —
-   gates: ≥48 gram files present, bf16 container already deleted
-   (disk), ≥146 GiB free, output ≥108,000 MiB, `rad-info` shows the
-   gptq receipt. No `--reuse`. Measured: 5 h 50 m (6.0–6.5 min/layer on
-   49 GPTQ layers, load-average 8), output **116,416 MiB**.
-   The plugin's `rad_arch_declare` enforces ONE ENCODING PER EXPERT
-   TABLE (`one table of experts takes one encoding, and only a whole
-   expert kept plain bf16 is served beside it`) — a split tier
-   (down=fp8, gate_up=w4) is unservable: the kernel serves a single
-   w4nl64a8h table; both expert tables ride GPTQ w4 (down calibrated,
-   gate_up uncalibrated), protected experts bf16 beside them.
-
-## The recipe tiers (scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe)
-
-| Tier | Encoding | Why |
-|---|---|---|
-| MoE experts, both `gate_up` and `down` (one w4 table/layer) | **GPTQ w4nl64a8h** — u4 codes, fwht128 rotation, fp8 scales/64 + f32 scale plane | the campaign tier: GPTQ on down-input Hessians from the 10M-token corpus (the tap rides the down projection; gate_up shares the table uncalibrated — one-encoding-per-table plugin law) |
-| 24 derived protected experts (whole) | **bf16** | energy-share law (step 5) |
-| shared expert + trunk attention/SSM linears | **int8**, scale per 128 cols, searched | trunk precision law |
-| hyper-cond + MTP head | **fp8_e4m3 row-groups** | hc+mtp fp8 law |
-| lm_head | **int8** | i8 lm_head law (E4M3 costs 0.6 top-1 alone) |
-| PLE n-gram table | fp8 at the checkpoint's fixed scale 1.9931793212890625e-04 | the fixed-scale law; absmax audit passed the 2⁻¹³..2⁻¹⁰ plateau |
-| embeddings, vision tower, QSA indexer | **bf16 passthrough** | untouched |
-
-## The shape (`scripts/frank-archon.env`)
-
-TP=2 + `wht6`, `--max-model-len 200000`, MNBT 2048 (the MTP resonance law —
-2560 collapses decode to 2.6–7.7 t/s), `--ngram-placement ram`, froggeric
-v22.5 template override. `MODEL_FILE` points at
-`/home/dank/models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416 MiB
-measured**); the serve size floor `FRANK_ARCHON_RAD_MIN_MIB` moves
-177000 → **110000**. Concurrency and the KV cache moved on 2026-10-11; that
-change is documented in its own section below.
-
-## 2026-10-11 — unquantized KV at 2 × 200,000
-
-The ask: two concurrent 200,000-token conversations on exact (unquantized)
-attention state. The fp8 pool backed 3,543 MiB → **386,904 tokens = 1.93
-sessions of the served window**, so two full-height conversations overflow it
-and the scheduler preempts the overflow to the host/disk prefix tiers
-(re-prefilling on return). Measured, that overflow is survivable — the fp8
-shape passed the same 2 × 197k pair probe — so this is a **headroom** change,
-not the repair of a hard failure. The hard limit the engine states is per
-request, not per pair: a request longer than the pool's token count is
-*"accepted by the API and then refused as too large to ever fit — this is a
-hard limit, not a backing ratio"*, and at 200,000 ≤ 386,904 no single request
-ever hit it.
-
-### The dtype law
-
-`--kv-cache-dtype` takes **only `bf16 | fp8`**: one `kv_dtype` for the whole
-cache. llama.cpp's exact-K / quantized-V pair (what frank-zealot runs as
-`--cache-type-k f16 --cache-type-v q8_0`) has **no radiance equivalent** — the
-rung under bf16 is fp8 on both halves. Evidence: the flag table in the image
-itself (`--kv-cache-dtype takes bf16 or fp8, not '%s'`) and the absence of any
-`cache-type-k`/`-v` symbol in the binary.
-
-bf16 costs **1.89× fp8, not 2×**, and the reason is in the group table:
-
-| Group | fp8 page | bf16 page | cost per token, per rank |
-|---|---|---|---|
-| `kv_attn` — 13 full-attention layers, 4-token blocks | 26.00 KiB | 52.00 KiB | 6.5 → 13.0 KiB |
-| `kv_qsa_bkey` — 13 layers, the QSA indexer keys | 3.25 KiB | **3.25 KiB** | 0.8125 KiB, dtype-fixed |
-| `kv_gdn_state` 54 MiB × 2 copies per sequence (+ conv/tail/ple) | per slot | per slot | ≈ 110 MiB **per sequence** |
-| prefix checkpoints 8 × 55.99 MiB (inside the pool) | 448 MiB | 448 MiB | fixed |
-
-→ `pool_mib ≥ tokens × 13.8125 KiB + 110 × seqs + 448`. The formula reproduces
-both live boots: fp8 `(3543 − 330 − 448) → 387,240` vs the ledger's 386,904;
-bf16 `(6912 − 220 − 448) → 462,946` vs the ledger's **462,752**. It is encoded
-as a test seam (`the KV pool backs seqs x max-model-len at the declared cache
-dtype`) so a future resize cannot silently under-provision the pool.
-
-### Why the weights slab pays — and why that is not pure loss
-
-The two VRAM budgets are one budget, and the ceiling is the engine's own — not a
-hand-picked reserve. Card 32,624 MiB, and the table that prints at the pair
-24,000/5,120 says `3.42 GiB is neither budgeted nor accounted — the driver's
-context, the activation arena and --gpu-headroom-mib come out of it; anything
-past those is throughput on the floor`, with `card 0 settled at 2.01 GiB free
-against 209.19 MiB (the 96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the
-static region)`. Claimable reserve above the ENGINE's floor:
-2,058 − 209 = 1,849 MiB ⇒ **`weights + kv ≤ 30,969 MiB`**. (The 29,120 bound
-written on 2026-10-11 was my own 1 GiB "keep free" reserve, measured off a card
-figure of 31,856 MiB; the engine never asked for that reserve and the seam now
-derives the number from its table.) Paying for a 6,912 MiB pool therefore cuts
-the resident slab 26,071 → 22,208 MiB (−14.8 %), and the evicted experts spill
-into the pinned host pool (40,960 → 44,928 MiB per rank; ledger: pinned pool
-slots 7,190 → 10,375). The pinned-RAM ledger rises to ≈147 GiB, so the TTM
-floor moved 140,000 → 150,000 MiB (the kernel cap is 204,800).
-
-Half of that loss is refunded by an engine behaviour worth knowing: **idle KV
-VRAM is the expert cache's spillover**. The fp8 pool lent 2.69 GiB to the expert
-slab; the bf16 pool lends **6.08 GiB** (`mover: 5146 slots can be lent out of
-the KV cache`). On a llama.cpp endpoint KV is inert storage; on radiance it is
-a competing cache, and every KV resize is also an expert-residency resize.
-
-### seqs 3 → 2
-
-`--max-num-seqs 3` was never a radiance requirement: it is the tcclaviger vLLM
-fork's torch.compile constraint (`ConstraintViolationError: Constraints
-violated (L['query_start_loc'].size()[0])`), retired with that runtime — see
-[[dankserver-qwen38-fp8-expert-offload-tuning]]. Radiance booted seqs 2 in the
-fp8-native era, and the 2026-10-06 decode sweep measured seqs 2 and 3 tied
-inside the ±3 % noise floor. Two slots × the window is the capacity contract;
-it also saves 110 MiB of per-slot recurrent state and drops the fused TP
-all-reduce to 2 × 3 = 6 rows, i.e. deeper inside the 25-row bound at which the
-wht6 wire is bit-exact.
-
-### Boot ledger (deployed 2026-10-11, commit `0bf26b0`)
-
-```text
-kv cache           6.75 GiB   349% of it -- backs 2 sequences all at 200000
-                                tokens at once, which is the worst case
-vram_weights      21.69 GiB   stated by --vram-weights-mib
-vram_kv            6.75 GiB   stated by --vram-kv-mib
-unclaimed          2.03 GiB
-kv_attn      full 13 layers block 4 tok page 52.00 KiB 115688 blocks = 5.74 GiB (462752 tokens)
-kv_qsa_bkey  full 13 layers block 4 tok page  3.25 KiB 115688 blocks = 367.17 MiB (462752 tokens)
-load: rank 0, mover took 21.69 GiB, weights pool 147.00 KiB free of 21.69 GiB
-mover: 5146 slots can be lent out of the KV cache (6.08 GiB)
-card 0 settled at 1.85 GiB free
-serving on http://:::7770
-```
-
-### Verification (live service, no scratch container)
-
-| Check | Result |
-|---|---|
-| `/health`, `/v1/models` | 200 `{"status":"ok","model":"frank-archon"}`; served id `frank-archon` |
-| 2 × ~197k concurrent, a unique marker buried at the head of each | **both answered their own marker exactly** — 196,919 + 196,918 = 393,837 tokens resident, above the fp8 pool's 386,904 and inside bf16's 462,752; walls 138.1 s / 66.2 s, `finish_reason: stop`, prefill 2,747 / 2,991 t/s |
-| decode at 51.5k depth, 512 tokens, greedy | 94.1 cold / 102.3 warm / 117.6 on a prefix-cache hit t/s; draft acceptance 0.7284 / 0.7525 (fp8 control at cutover: 0.7246) |
-| fp8 control arm, same probes (`scripts/probe-frank-archon-kv.py`) | table in the next section |
-
-Probe law re-learned, worth recording: radiance routes the thinking block to
-`reasoning_content`, so a chat request whose budget is spent inside `<think>`
-comes back with **empty `content` and `finish_reason: stop`** — a probe that
-forbids thinking (`chat_template_kwargs.enable_thinking=false`) and reads both
-fields is the only honest shape. The first pair probe scored 0/2 on exactly
-that artifact, not on a KV fault.
-
-### Measured cost — A/B on the live service
-
-Method (no scratch container, no mask — the unit file is a relink symlink and
-`mask` refuses it): write a variant env file on the host
-(`/home/dank/frank-archon-fp8control.env`), point a transient drop-in at it
-(`Environment=FRANK_ARCHON_ENV_FILE=…`; drop-ins are read AFTER the unit's own
-`Environment=` lines, so they win), `daemon-reload`, restart, probe, delete the
-drop-in, restart. Control = the deployed 2026-10-08 shape, verified from
-`podman inspect` argv (`--kv-cache-dtype fp8 --vram-weights-mib 26071
---vram-kv-mib 3543 --max-num-seqs 3`) and the ledger page size (26.00 KiB).
-Identical probes, same salts, loopback, greedy, 512 generated tokens.
-
-| Probe | fp8 control (26071/3543, seqs 3) | bf16 deployed (22208/6912, seqs 2) | Δ |
-|---|---|---|---|
-| prefill 51.5k, cold | 4,379 t/s | 3,126 t/s | **−29 %** |
-| prefill 51.5k, warm | 4,864 t/s | 3,333 t/s | **−31 %** |
-| decode 51.5k, cold | 124.1 t/s | 94.1 t/s | −24 % |
-| decode 51.5k, warm | 99.2 t/s | 102.3 t/s | +3 % |
-| decode on a prefix-cache hit | 127.7 t/s | 117.6 t/s | −8 % |
-| draft acceptance, mean of passes | 0.661 | 0.740 | **+12 %** |
-| 2 × 196,919 concurrent pair | both markers HIT, 110.5 s | both markers HIT, 138.1 s | +25 % wall |
-
-Read it as: **the price of exact KV on this engine is prefill, not decode.**
-Prefill is expert-bandwidth-bound and the resident slab is 14.8 % smaller, so
-time-to-first-token at the full window rises ~25 % (the pair: 110 s → 138 s);
-both prefill passes agree, far outside the ±2 % prefill noise floor. Decode is
-inside the ±3 % band on two of three passes (warm +3 %, hit −8 %); the cold-pass
-−24 % is the largest single number, the least reproducible, and it moves the
-same direction as prefill — the cold arm is paying expert streaming, not
-attention. Spec-decode acceptance IMPROVED ~8 points (0.661 → 0.740): exact
-attention state makes the target agree with the MTP drafter more often, which
-pays on every token of every conversation, and it is the strongest argument for
-the shape.
-
-Open call (Frank): keep exact KV at the prefill cost, or take the cheap fp8
-variant — `--vram-kv-mib 3,648` + `--max-num-seqs 2` fits 2 × 200,000 at fp8
-for a 105 MiB pool bump and ≈ no decode cost. The tool is committed:
-`scripts/probe-frank-archon-kv.py` (run it on Dankserver; it rebuilds its
-prompts from the repo checkout there).
-
-### Rollback
-
-Revert `scripts/frank-archon.env` to the fp8 pair (26071/3543, fp8, seqs 3,
-host pool 40960, prefix host 4096) and redeploy: no payload change, boot ~2 min.
-The KLD gate scored at the fp8-KV shape, so a bf16-KV production runs *away*
-from the scored configuration — bf16 can only reduce the attention-state
-error, and a re-score is a drained-window job if the gate agreement is ever
-needed in the record.
-
-## 2026-10-13 — the power failure, the ttm reset, and 4 × 200,000 fp8 KV
-
-### The outage: a refusal, not a crash
-
-Dankserver lost power on 2026-10-12 ~23:17. The box came back; frank-archon
-never did. The unit sat in `activating (auto-restart)`, exiting 1 about 50 ms
-after starting, with one journal line on repeat:
-
-```text
-ERROR: ttm pages cap is 125988 MiB, the pinned ledger needs >= 150000
-frank-archon: raise /sys/module/ttm/parameters/pages_limit …
-```
-
-**6,919 refusals in ~20 hours.** No container was created (`podman ps -a` showed
-the name with a zero start time), so there was no engine, no port, and no engine
-log — the signature of a preflight refusal, which every deployer/monitor view
-outside the journal renders as an unexplained crash loop.
-
-The cause is a class, not an incident. The pinned-RAM ledger (~147 GiB: 2 × 44 GiB
-expert pools + the 47.7 GiB PLE table + the 8 GiB prefix host tier) rides TTM's
-shared pages, capped by `/sys/module/ttm/parameters/pages_limit`. **That knob is
-runtime state, and its default is half of RAM** — so the manual raise of
-2026-10-05 (52,428,800 pages = 200 GiB) died with the power failure and the cap
-fell back to 32,253,039 pages = 125,988 MiB, 24 GiB below the floor the serve
-script enforces. The fp8-native era had planned the persistent half of this
-("add `ttm.pages_limit=52428800` to the kernel cmdline at the next scheduled
-reboot", [[frank-archon-radiance-qwen38-native-fp8]]); it was never applied, and
-a power failure is not scheduled.
-
-### The repair: the gate raises the knob itself
-
-`scripts/ttm-preflight.sh` (sourced by `serve-frank-archon.sh`) reads the cap as
-the MIN over the knobs the running kernel exposes, and when it is below
-`FRANK_ARCHON_TTM_MIN_MIB` it writes `FRANK_ARCHON_TTM_TARGET_MIB` (204,800 MiB =
-52,428,800 pages) into every writable knob, **re-reads**, and only then refuses.
-The unit is root and the knob is a 0644 root-owned module parameter, so the write
-is legal; a kernel that refuses the move produces the same named refusal as
-before, and a reboot is now a boot rather than a visit to the machine.
-
-| Knob | Power-failure default | After the gate |
-|---|---|---|
-| `/sys/module/ttm/parameters/pages_limit` | 32,253,039 pages = 125,988 MiB | 52,428,800 pages = 204,800 MiB |
-
-Seam-tested, not inspection-tested: *the ttm gate raises a reset pages cap and
-refuses when the write fails* and *the ttm cap counts only the knobs the kernel
-exposes* drive the gate against a temp knob directory — a below-floor writable
-knob ends at the target and the start passes; a read-only knob leaves the cap
-untouched and the start refuses by name; an absent `pages_max` is skipped, never
-read as zero (a zero reading would refuse a healthy host).
-
-### The shape: 4 slots × 200,000, fp8 KV (same day, superseded by the re-cuts below)
-
-Answers the open call of 2026-10-11 from the other side: Frank asked for **four**
-concurrent 200,000-token conversations with quantized KV.
-
-fp8 is not a preference at four slots, it is the only dtype that affords them.
-The pool must back `seqs × max-model-len` (the engine's hard limit), so the
-fourth slot is bought in KV, out of the one per-card budget
-(`weights + kv ≤ 29,120 MiB`, the bound as it was understood then):
-
-| dtype | pool for 4 × 200,000 | weights slab left |
-|---|---|---|
-| bf16 | 10,791 + 440 + 448 = 11,679 MiB | 17,441 MiB (−33 % vs the fp8-era 26,071) |
-| **fp8 (deployed)** | 5,713 + 440 + 448 = 6,601 MiB | **22,208 MiB — unchanged** |
-
-So `--vram-weights-mib 22208` / `--vram-kv-mib 6912` stay exactly where the
-2026-10-11 change put them (pair sum = the measured 29,120 ceiling), the
-44,928 MiB/rank pinned host pool is untouched, and the RAM/TTM ledger does not
-move. What the fp8 pool buys: `(6912 − 110×4 − 448) × 1024 / 7.3125` =
-**843,566 predicted tokens** = four full-height conversations with 5.4 % slack
-(the bf16 pool backed 462,752 = two; the 2026-10-08 fp8 pool backed 386,904 =
-1.93).
-
-Terms that move with the dtype and the slot count:
-
-- **Per-slot recurrent state** is dtype-independent and lives inside the pool:
-  +110 MiB per slot, 440 MiB for four (already in the arithmetic above).
-- **The fused TP all-reduce** is declared over `seqs × (1 + spec)` rows: 4 × 3 =
-  **12 rows**, still inside the 25-row bound at which the `wht6` wire is
-  bit-exact — so four slots cost nothing in wire exactness.
-- **Prefix-cache entries shrink** with the attention plane (110.50 KiB at bf16 →
-  58.50 KiB at fp8), so the 8 GiB host tier parks ~5.8 sessions of 200k where at
-  bf16 it parked ~2.9 — four live slots plus a parked set fit with no re-size.
-- **`models.yml` is untouched**: `contextWindow` quotes the per-request window
-  (200,000), which did not change; only the concurrency did.
-
-Expectation carried from the 2026-10-11 A/B, whose arms bracket this shape
-(same slab 22,208): prefill is set by the slab, not the dtype, so ~3.1k t/s
-cold / ~3.3k warm; decode sits in the 94–118 t/s band with draft acceptance
-~0.73–0.75. The 4,379 t/s fp8 prefill of the 2026-10-08 control belongs to the
-26,071 MiB slab and does not come back by quantizing KV.
-
-### Boot ledger (2026-10-13 19:21, four slots, fp8 — superseded same evening)
-
-The sizing law's prediction, checked against the engine's own arithmetic:
-predicted 843,566 tokens, ledger **843,276** (0.03 % block rounding).
-
-| Group | Page / state | Size | Tokens |
-|---|---|---|---|
-| `kv_attn` | 13 layers, 4-token blocks, **26.00 KiB** (fp8 page) | 5.23 GiB | **843,276** |
-| `kv_qsa_bkey` | 13 layers, 4-token blocks, **3.25 KiB** (dtype-fixed) | 669.10 MiB | **843,276** |
-| `kv_gdn_state` | 36 layers, 54.00 MiB, **8 slots** (4 × 2 copies) | 432.00 MiB | per-slot |
-| `kv_gdn_conv` / `kv_ple_*` | 36 + 1 + 1 layers, 4 slots | 7.91 MiB | per-slot |
-
-`kv cache 6.75 GiB … backs 4 sequences all at 200000 tokens at once, which is
-the worst case` — the capacity contract, in the engine's words. The rest of the
-boot is the fp8-era picture: `vram_weights` 21.69 GiB filled to 147 KiB (rank 0)
-/ 355 KiB (rank 1) of slack, `unclaimed` 2.03 GiB, cards settled at 1.99 /
-2.00 GiB free, `mover: 4967 slab slots (64 shadow), 10375 pinned pool slots,
-4.24 GiB static vram`, and the half-refund in plain sight: `4967 slots can be
-lent out of the KV cache (5.87 GiB)` (the fp8-era 3,543 MiB pool lent 2.69 GiB).
-
-Two ledger lines that confirm the derived consequences: the sampler prints
-**12 rows** (4 sequences × (1 + 2 speculative) — inside the 25-row bound where
-`wht6` is bit-exact), and the prefix/tier arithmetic runs on the fp8 entry size
-(58.50 KiB, half of the bf16 entry).
-
-### The same-day re-cuts: 4 → 3 slots, then the slab takes the card's slack
-
-Frank's call, twice in one evening. Both are one-env-pair changes, and both were
-read off the boot ledger rather than assumed.
-
-**4 → 3 slots (22208/6912 → 24000/5120).** Three slots need
-3 × 200,000 at fp8 = 4,285 + 330 + 448 = **5,063 MiB** of pool; the 6,912 pool
-was backing 858,969 tokens — 43 % more window than the contract asks for.
-Cutting it to 5,120 (predicts 608,027; ledger **607,740**, 0.05 %) hands
-1,792 MiB to the slab and drops the all-reduce to 9 rows.
-
-**24,000 → 25,600 (the card's settling slack).** The 24,000 boot printed
-`card 0 settled at 2.01 GiB free against 209.19 MiB (the 96.00 MiB
---gpu-headroom-mib and 113.19 MiB kept for the static region)` — 1,849 MiB above
-the ENGINE's own floor, with the budget table calling the surplus `throughput on
-the floor`. Claiming 1,600 MiB of it puts the pair at 30,720 and the ledger at
-`unclaimed 477.30 MiB` (cards ~336 MiB free, floor 209.19).
-
-| pair | slab slots (rank 0) | pinned pool slots | A-cold 43.5k | concurrent 197k |
-|---|---|---|---|---|
-| 22208/6912, seqs 4 | 14,753 | 10,375 | 3,397 t/s | 3,307 t/s (4-way) |
-| 24000/5120, seqs 3 | 16,235 | 8,893 | 3,290 t/s | 3,051 t/s (3-way) |
-| **25600/5120, seqs 3** | **17,548** | 7,580 | **3,584 t/s** | **3,589 t/s** (3-way) |
-| 26071/3543, seqs 3 (2026-10-11 control) | — | 7,190 | 4,379 t/s | — |
-
-The claim shipped with the 25,600 arm — that the slab is a THRESHOLD below
-~26 GiB so the extra 1.6 GiB buys ~0 — is **falsified by its own measurement**:
-+8.9 % single-stream, +17.6 % concurrent, both outside the ±3 % noise floor. Two
-slab points that land inside noise of *each other* (22,208 vs 24,000) are not a
-plateau; the pinned host pool is the slow path, so every MiB moved from RAM onto
-the card is throughput (the ledger shows the transfer directly: slab
-+1,313 slots, pinned pool −1,313).
-
-The cliff is real, it just sits above 25,600: 26,071 measured 4,379 t/s, +22 %
-for +471 MiB. At three slots that is out of reach (26,112 + 5,120 = 31,232 > the
-30,969 ceiling). At **two** slots the pool falls to 3,072 and the slab reaches
-**27,897** — the ceiling exactly, slab above the cliff. That arm is untested and
-it is a capacity decision, not a tuning one.
-
-Capacity contract, live-verified on the 3-slot shape: three concurrent prompts
-of **199,984 tokens** each (599,952 tokens) were accepted and completed on
-`/v1/completions`; a fourth full-height request is refused as "too large to ever
-fit" — radiance never truncates.
-
-### The decode ladder (2026-10-13): slots are paid in concurrency, not prefill
-
-Frank's challenge — "why are you not tracking decode?" — was right, and the
-answer is a number now. 512 generated tokens (the bench law), greedy, cold
-prompts, this shape (3 x 200,000, fp8, slab 25,600):
-
-```
-1-stream  43.5k: decode  99.0 t/s  (prefill 4,042, accept 0.952)
-1-stream   197k: decode  93.3 t/s  (prefill 4,826, accept 0.938)
-3-stream  43.5k: decode 113.9 / 43.3 / 10.9 t/s   wall 83.4s
-3-stream   197k: decode  88.9 / 16.3 / 64.6 t/s   wall 122.4s
-```
-
-Two laws came out of it. **Decode is nearly window-free** — 99.0 at 43.5k,
-93.3 at 197k, −6 % for 4.5x the depth — so a long conversation costs
-prefill, not tokens-per-second. **Concurrency is where the slot count is
-paid**: three simultaneous streams give 18.4 t/s aggregate at 43.5k and 12.5
-at 197k against 99 single-stream, and the split is a lottery
-(113.9/43.3/10.9), not a fair share — the fused-step schedule hands the step
-to whoever it reached first, so one stream finishes at 113 t/s while its
-sibling crawls at 10.9. That is what "3 slots" owes per token, and no prefill
-number shows it.
-
-Corpus caveat, stated at the claim: this probe's filler is a repetitive
-ladder, so MTP acceptance is 0.93–0.97 against the 0.740 of the 2026-10-11
-table below; these rows flatter decode relative to that table and are the
-baseline for *this* corpus, not a replacement for those rows.
-`probe-frank-archon-kv.py` now carries this leg as **D** (leg B generates 24
-tokens, which is why a day of shape work priced only prefill).
-
-### Recovery evidence
-
-```text
-Oct 13 19:21:08 dankserver ttm-preflight.sh[1623778]: frank-archon: ttm pages cap
-  is 125988 MiB, below the pinned-ledger floor 150000 — raising the knob to
-  204800 MiB (52428800 pages)
-Oct 13 19:21:08 dankserver ttm-preflight.sh[1623778]: frank-archon: ttm pages cap
-  204800 MiB clears the pinned-ledger floor 150000
-Oct 13 19:21:08 dankserver systemd[1]: Started frank-archon.service
-```
-
-`cat /sys/module/ttm/parameters/pages_limit` → 52,428,800 (204,800 MiB) after
-the boot, i.e. the root `ExecStartPre` hook did the write the service process
-(User=dank) cannot.
-
-### Capacity probe (2026-10-13, live service)
-
-`scripts/probe-frank-archon-kv.py` now reads the slot count, window and KV dtype
-from the deployed env on the host, so the probe cannot drift from the service:
-it printed `shape: 4 x 200,000 tokens, kv fp8` and fired **four concurrent chats
-of 196,960 prompt tokens each** (646,299 chars, 98.5 % of the window), each
-carrying a marker whose first word is unique.
-
-| Stream | prompt | wall | prefill | retrieval |
-|---|---|---|---|---|
-| B1 | 196,961 | 238.3 s | 1,423 t/s | exact, HIT |
-| B2 | 196,960 | 98.0 s | 2,043 t/s | exact, HIT |
-| B3 | 196,961 | 201.1 s | 993 t/s | exact, HIT |
-| B4 | 196,960 | 149.9 s | 1,327 t/s | exact, HIT |
-| **total** | **787,842** | **238.4 s** | **3,305 t/s aggregate** | 4/4 |
-
-All four answered their own marker; nothing was preempted, refused, or truncated
-(`finish=stop`, `cache_n=0`, so no prefix cache padded the result). Aggregate
-prefill equals the single-stream rate measured in the same probe (3,397 t/s at
-43.5k), i.e. the four streams share one prefill pipeline at the slab-limited
-ceiling and the engine never ran short of KV. Prefix cache intact: the same
-43.5k prompt repeated answered in 1.0 s with `cache_n=43008`.
-
-The per-stream `decode` column is NOT a decode rate here — 8 generated tokens
-make it first-token dominated (the bench law: generate ≥ 128 tokens to read a
-rate). This probe answers one question only: does the endpoint owe four
-full-height conversations at once, and answer each one exactly. It does.
-
-## Gates (measured at cutover)
-
-- **KLD gate: PASS (2026-10-08 04:12)** — `kld-gate-swift15.sh`,
-  candidate on 1.0.13 vs `kld-ref`: **mean KL 0.0146** (median 0.00126,
-  p90 0.0315, p99 0.194, max 4.12), **top-1 97.84 %**,
-  **ppl ratio 1.0126** (candidate 1.6913 vs reference 1.6702), 50,235
-  positions / 197 docs. Report: `/home/dank/radiance-state/kld-gate.json`
-  (+ `.rows` per position). Gate shape: disk-tier, pool 40960, MNBT 3072
-  (the w4 activation arena scales hard with MNBT: 32768 → 33.3 GiB arena,
-  refused), KV 2048, headroom 512, no speculation. The engine ran rank 0
-  fully on card 0 (25.88 GiB) with rank 1 host-side in this shape —
-  scoring is placement-invariant; scored at 28 positions/s.
-- **KLD gate re-PASS (2026-10-08 04:49) at the production-legal shape**:
-  MNBT 2048 (the w4 file-tier law: experts stream in place only at
-  MNBT ≤ 2048 — the recording's 32768 demands a widened form the disk
-  tier refuses), pool 40960, vram-kv 3072 fp8, tp 2, headroom 768.
-  **mean KL 0.0131, top-1 97.92 %**. The Δ vs the first PASS (0.0015) is
-  the float-order noise band of the chunk-size change — the verdict is
-  chunk-size invariant, ~2.3× inside the 0.03 bar on both shapes.
-- **Throughput gate: PASS (2026-10-08 05:40)** — decode at 44.5k depth
-  (streamed chat probe, temp 0, 512 tokens, salted): arm A (main serve
-  shape, EVC 0.82) **146–173 t/s**, engine timings **121.9 t/s**,
-  draft accept **0.7246** (350/483); arm C (deployed vram-pair shape
-  26071/3543) median **136.0 t/s**, warm TTFT 9.45 s. Control
-  (fp8-native 1.0.13): 30.08 / accept 0.7549. Prefill ~4.7k t/s warm.
-  The w4 expert table is resident — the fp8-native era's NVMe expert
-  streaming (the 3.5 GB/s wall) is gone; the MTP ×3 head rides a 45–60
-  t/s base decode. Conversation battery (short context): geomean
-  172 t/s (min 100 / max 208), mean ttft 0.09 s. The EVC ladder is moot:
-  the w4 resident set fits fully inside the 40 GiB pool at 0.82
-  (ledger: mover full fill). Depth-output audit: the 44.5k-depth
-  summary is a coherent ordered index of the concatenated corpus
-  (no repetition, no NaN-token garbage) — the depth-corruption class
-  of 2026-10-04 is clean.
-- Fleet probes: PASS, re-run on the deployed prod — vision (solid-red
-  512 px → "Red"), tool_calls (`{"expr":"23 * 47"}`), reasoning_effort
-  xhigh/medium/low/high all 200 (the Swift1.5 froggeric template accepts
-  the full set), reasoning_content present, response_format json_object
-  honoured. **Vision probe law:** budget ≥ 256 tokens and read
-  `reasoning_content` as fallback — at 64 tokens thinking exhausts the
-  budget and `content` comes back empty (the answer lands in the
-  reasoning channel).
-- **Deployed verification (2026-10-08)**: `deploy.sh dankserver
-  --service frank-archon` green incl. Playwright e2e 33/33; unit
-  `active`, `/health` 200, `/v1/models` id `frank-archon` (the harness
-  alias is the served-model-name, independent of the file — models.yml
-  untouched). Deployed shape: fp8 KV (KLD-gate shape), spec depth 2,
-  seqs 3, vram pair 26071/3543. Boot ledger: KV pool 3.46 GiB, expert
-  pools 57.84 GiB, cards settled 1.41 GiB free, `serving on :7770`.
-  Live decode probe (512 tokens, streamed, temp 0): ~190 t/s.
-
-### Wire rename + the bench drop-in trap (2026-10-09)
-
-`FRANK_ARCHON_SERVED_NAME` moved the served id `frank-primary` →
-**frank-archon** (commit `7c77326`, 2026-10-09); the service, unit, scripts and
-env file kept `frank-primary` until the 2026-10-10 rename (`55f87fc`) brought
-them to `frank-archon` too. Radiance is the only
-fleet engine whose `/v1/models` listing reflects the flag, and it echoes the
-**requested** model name in chat responses — so a chat probe that returns 200
-proves nothing about the served id. Always read `/v1/models`.
-
-The first deploy after the edit came up green and `/v1/models` still said
-`frank-primary`. Cause: a campaign leftover drop-in
-`/etc/systemd/system/frank-archon.service.d/bench.conf` (2026-10-08 04:09)
-overrode BOTH `Environment=FRANK_ARCHON_ENV_FILE` and `ExecStart` to the
-staging worktree `/home/dank/serverAppConfigs-swift15`. Every
-`deploy.sh dankserver --service frank-archon` since then restarted the
-staging tree: green health gate, old configuration, silently. Verified before
-removing: the staging `serve-frank-archon.sh` and `frank-archon.service`
-are byte-identical to the checkout, its `frank-archon.env` differed only by
-the alias line, and its HEAD (`8375567`) is an ancestor of `main`. The
-drop-in was staged in `/home/dank/.AGENTTRASH/20261009-frank-archon-bench-dropin/`
-(that dated dir is gone — the trash dir is empty on Dankserver as of 2026-10-18).
-After `daemon-reload` + restart, `/v1/models` reports `frank-archon` (boot
-healthy in 102 s, shape unchanged: fp8 KV, spec depth 2, seqs 3, vram pair
-26071/3543).
-
-**Deployer blind spot:** `deploy.sh` plans from repo paths and gates on the
-port. It cannot see host-side drop-ins, so unit drift is invisible to a green
-run. After an LLM deploy, verify the served configuration behaviourally
-(here: `podman inspect <name> --format {{json .Config.Cmd}}` and `/v1/models`).
-
-
-
-## Deploy + verify
-
-```bash
-sudo systemctl stop frank-archon          # drained window
-bash convert-swift15-w4dn8.sh              # if not already run
-./scripts/deploy.sh dankserver --service frank-archon
-```
-
-Post-cutover checklist: unit active/enabled; `/health` 200; boot ledger
-(expert pool coverage now w4-sized, KV pool should be larger than the
-FP8-native era's 399,824 tokens); MTP draft counts in `timings`; froggeric
-probe; ≥256 px `image_url` vision probe answers the color; tool-call
-probe; `reasoning_effort` fold; Prometheus target green; `tests/run.sh`
-+ Playwright e2e green.
-
-## Rollback
-
-| Layer | Path |
-|---|---|
-| Publisher iq4r (successor) | [[frank-archon-radiance-qwen38-iq4r]] — the LIVE payload since 2026-10-15 |
-| FP8-native radiance | rebuild-only — the `.rad` (173.0 GiB) was `rm`'d 2026-10-15: `scripts/download-frank-archon.sh` + `scripts/build-frank-archon-rad.sh` |
-| FreeToken PR #587 | git tag `frank-archon-freetoken-pr587-20261005` |
-| tcclaviger vLLM shape G | git tag `frank-archon-vllm-shapeG-20261003` |
-
-**This era's own payload** (`qwen4exp-swift15-w4dn8-i8-hc8m.rad`,
-116,416 MiB) was purged in the 2026-10-17 disk cleanup (trash-staged, then
-emptied with Frank's approval). The byte-exact container is published as
-`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`
-(`qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, Hub LFS sha256
-`09dd93064365009c508b59012adadab6a36791a50fa319f76b31bf359c01e9ff`) —
-re-download restores the exact payload; a full re-convert is no longer
-possible without re-fetching a BF16 source (see next paragraph).
-
-Campaign artifacts: the raw BF16 checkpoint was `rm`'d 2026-10-15 (Frank's
-one-time authorization), so re-convert needs a fresh BF16 source; `kld-ref`
-(24 GiB), grams (`/home/dank/corpora/calib`), corpora JSONL, and the logs
-`/home/dank/logs/swift15_*.log` remain. The BF16 `.rad` container was
-deleted before the final convert (disk gate).
-
-## Links
-
-- [[frank-archon-radiance-qwen38-native-fp8]] — the replaced FP8-native era
-- [[LLM/index]] — fleet index
-- [[swift15-campaign-worklog]] — campaign record
-- [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here

serverAppConfigs-docs/LLM/frank-archon.md

@@ -0,0 +1,419 @@
+---
+type: reference
+title: "frank-archon on Dankserver — radiance (StillDeadcode) 1.0.13, publisher Qwen3.8-Flash-Next iq4r .rad, 2× R9700 TP=2 (ACTIVE)"
+description: "The one frank-archon page: the live radiance shape and harness contract (read off scripts/frank-archon.env), the Eras table carrying every rollback path including the re-download doors for the purged payloads, the laws this endpoint taught (pinned-RAM/TTM preflight, the KV dtype law, the one per-card VRAM budget, descriptor selection), the wire contract, and the files that own it. It replaces the nine per-era runbooks."
+tags:
+  - llm
+  - frank-archon
+  - dankserver
+  - radiance
+  - gfx1201
+  - r9700
+  - tp2
+  - expert-offload
+timestamp: 2026-10-18
+---
+
+# frank-archon — radiance (StillDeadcode), 2× R9700 TP=2, Dankserver :7770
+
+The radiance engine (C++/HIP, `libr4d` kernels built for gfx1201, no Python in the serving
+path) driving `podman run` straight from the digest-pinned image under a **root** systemd
+unit; container name = unit name. Host Dankserver (192.168.0.235), port 7770, both Radeon AI
+PRO R9700s. Capacity contract **3 conversations × 200,000 tokens at fp8 KV**; a payload swap
+is a `MODEL_FILE` line plus a re-download — see Eras.
+
+## Current shape
+
+Authoritative source: `scripts/frank-archon.env` + `scripts/serve-frank-archon.sh` (the env
+header carries the measured arithmetic beside each number).
+
+**Payload** — `/home/dank/models/rad/qwen3.8-next-flash-fp8-iq4r-moe.rad`, 121,969,901,568 B
+= **116,322 MiB**, sha256 `0af5e962…` (the Hub LFS oid), from
+`StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe`: the engine author's own rad-convert of the
+**official (censored)** `Qwen/Qwen3.8-Flash-Next` bf16 checkpoint. Encodings (recipe
+`qwen4exp-w4nl64-i8-hc8m`): routed experts as 4-bit codes in a 16-level non-uniform codebook
+(128-pt Walsh–Hadamard rotation, GPTQ on a 10M-token calibration + 3 coordinate-descent
+sweeps, E4M3 scale per 64 weights); **10 whole bf16-protected experts per layer**; int8 W8A8
+trunk + lm_head; fp8 HC mixtures; PLE n-gram fp8 at the checkpoint's fixed scale (47.7 GiB);
+MTP head and the bf16 27-layer vision tower **in the container**. Card quality vs bf16: mean
+KL 0.0789 all-tokens / 0.0239 assistant turns, top-1 92.85 %. The abliterated family lives on
+[[frank-dragoon]]. Fetched by `scripts/download-frank-archon-iq4r.sh --fetch` (hfget resume +
+declared size + sha256) as the transient unit `iq4r-fetch` (log
+`/home/dank/logs/iq4r-fetch.log`, `…rc` sentinel); the staging dir
+`/home/dank/models/StillDeadcode/…` keeps card, LICENSE and recipe as provenance, and only
+`.rad` files are mounted.
+
+**Runtime** — `stilldeadcode/radiance@sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e`
+= pin **1.0.13** (Codeberg `v1.0.13`, commit `53282fa`): a 287 MiB image in dank's
+**rootless** storage, host needs nothing but the amdgpu driver. Pin the INDEX manifest digest,
+never the image ID (`69d05724…` is the config digest and sends `podman run` to the network).
+The serve script's gate is that digest ref itself — `podman image exists name@sha256:…` in one
+command (the 2026-09-30 no-pipe/grep-race lesson) — so a retagged look-alike or a newer
+`latest` cannot satisfy it. 1.2.0 is a rejected serve pin (Laws); the
+`--vram-weights-mib`/`--vram-kv-mib` pair is accepted by both pins, which is the rollback symmetry.
+
+**Trio** — `scripts/frank-archon.env` (every number, with its arithmetic in the header),
+`scripts/serve-frank-archon.sh`, `scripts/frank-archon.service`. Gates run in order: image
+digest present → `.rad` present and ≥ `RAD_MIN_MIB` (a truncated convert would otherwise buy a
+guaranteed long-timeout boot instead of this fast failure) → template present → `/dev/kfd` +
+`/dev/dri` → `ttm_ensure_cap` (the serve script re-verifies the cap after the unit's hook) →
+`podman rm -f` over `FRANK_ARCHON_LEGACY_CONTAINERS` (`frank-archon-qwen frank-archon-r9v
+qwen-fp8 glm-probe`) plus its own name, because systemd SIGKILLs conmon on a crash and leaves
+the name registered, and every one of those identities holds this GPU. Container contract
+(publisher-vetted, `docs/DOCKER.md`): `--network host`, `--init` (so the engine releases the
+cards on SIGTERM), `--security-opt seccomp=unconfined` (the default profile refuses io_uring —
+the PLE table's reads — and the NUMA calls the ROCm runtime makes for host memory), `--ulimit
+memlock=-1:-1`, `/models` ro + `/data` rw + the tune cache at `/opt/radiance/share/radiance/tune`,
+template as a single-file mount, then `exec podman wait`: the script's lifetime IS the
+container's, so a crash is a nonzero `ExecStart` and the restart policy is real.
+Unit: `After=network-online.target`, `WantedBy=multi-user.target`, `User=dank`,
+`LimitMEMLOCK=infinity`, `StartLimitIntervalSec=0`, `OOMScoreAdjust=-500` (the last process the
+OOM killer should pick), `Restart=always` **not** `on-failure` — a clean API-server exit after a
+worker fault reads as success to systemd (the 2026-10-01 vLLM lesson) — and
+`TimeoutStopSec=180` > the `ExecStop=podman stop -t 120` grace, because systemd's 90 s default
+cut the stop short and recorded `Failed with result timeout` (measured 2026-10-01 13:27:17). No
+drain contract is claimed: the OpenAI front end holds no queue, so an in-flight decode is lost
+on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_period`).
+
+| Knob (env name, `FRANK_ARCHON_` dropped) | Live value |
+|---|---|
+| TP / wire | `TP=2`, `TP_WIRE=wht6` — libr4d one-shot all-reduce, not RCCL; `min-kb` stays the engine default |
+| Slots / window | `MAX_SEQS=3`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) |
+| KV dtype | `KV_DTYPE=fp8` — the only dtype that funds 3 slots (Laws) |
+| VRAM pair | `VRAM_WEIGHTS_MIB=25600` + `VRAM_KV_MIB=5120` = 30,720 of the 30,969 ceiling; `GPU_HEADROOM_MIB=96` |
+| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=44928` **per rank** |
+| PLE | `NGRAM_PLACEMENT=ram` (47.7 GiB pinned; refuses boot on a short memlock) |
+| Spec / batch | `SPEC_TOKENS=2` — the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); the publisher's card ships depth 3 as its peak and this fleet's decode ladder was measured at 2. `MAX_BATCHED_TOKENS=2048` |
+| Prefix cache | `PREFIX_HOST_MIB=8192` + `PREFIX_DISK_MIB=65536` under `STATE_DIR=/home/dank/radiance-state` (mounted rw at `/data`) |
+| Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it |
+| TTM gate | `TTM_MIN_MIB=150000`, `TTM_TARGET_MIB=204800` |
+| Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) |
+
+Boot ledger for this shape (2026-10-15, pin 1.0.13): pool 5.00 GiB backs **607,740 tokens**
+(`kv_attn` fp8 page 26.00 KiB, `kv_qsa_bkey` 3.25 KiB), `kv_gdn_state` 54 MiB × 6 slots, slab
+**17,548 slots** with 4.22 GiB of KV lent to it, sampler **9 rows**, template 28,234 B, vision
+live (≤ 4,194,304 px, 16,384 patches/pass).
+
+The boot **ledger**, not this arithmetic, is the referee: radiance states every budget before
+anything is allocated (weights vs KV, the host pool's coverage of the worst case, the PLE pin,
+the P2P answer per pair) — read it after every resize, and unlike every predecessor there is no
+per-boot VRAM planner to second-guess. For comparison, the card's own suggested run line on 32
+GB cards is `--host-pool-mib 12288 --expert-vs-cache-ratio 0.82` with seqs 8 and spec 3; the
+shape above is the fleet's measured one. `FRANK_ARCHON_EXTRA_ARGS` is the boot-ladder escape
+hatch (unset → no extra argv; it is expanded unquoted on purpose, so keep multi-flag values as
+one string).
+
+**Harness contract** (`/home/blank/.omp/agent/models.yml`) — provider `llamacpp (dankserver)`
+→ `http://192.168.0.235:7770/v1`, `api: openai-completions`, `auth: none`, `id: frank-archon`
+(= `FRANK_ARCHON_SERVED_NAME`; the provider *label* is legacy and renaming it breaks
+`config.yml` references). `contextWindow: 200000` must equal `MAX_MODEL_LEN`;
+`maxTokens: 16384` (clamp `min(maxTokens, window − estimate − 4096)`; compaction fires at
+85 % → 170,000 est × 1.03 + 16,384 ≤ 200,000); `input: [text, image]`;
+`compat.reasoningContentField: reasoning_content`; `reasoningEffortMap` minimal/low→low,
+medium→medium, high/max→xhigh; `extraBody` temperature 1.0 / top_p 0.95 / top_k 20 / min_p 0 /
+presence_penalty 0 / repetition_penalty 1.0 + `chat_template_kwargs {enable_thinking,
+preserve_thinking}`, every key inside the engine's known-parameter list (unknowns 400).
+`compactionModel` routes to `llamacpp (dankclaw-3090)/frank-zealot`.
+
+**Deploy / verify** — `./scripts/deploy.sh dankserver --service frank-archon`; launch it
+detached when the swap restarts the endpoint your own session runs on. Then verify
+**behaviourally**, not by the green gate: `podman inspect frank-archon --format '{{json
+.Config.Cmd}}'` and `/v1/models` — `deploy.sh` plans from repo paths and gates on the port, so
+host-side unit drift is invisible to it.
+
+## Eras
+
+One row per retired era on this port. The **Rollback** cell is the load-bearing one: anything
+marked "re-fetch" is gone from `/home/dank/models/`, which today holds only `StillDeadcode`,
+`rad` and `tpl.jinja`.
+
+| Window | Payload | Runtime | Shape | Replaced because | Rollback |
+|---|---|---|---|---|---|
+| 2026-10-08 → 10-15 | in-house `qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB (Swift1.5 abliterated BF16, GPTQ w4 experts + 24 bf16 protected, KLD 0.0131 / top-1 97.92 %) | radiance 1.0.13, same digest | 3 × 200,000 fp8 KV, slab 25,600 / pool 5,120 — the shape the live file still carries | the engine author's own container is the supported path and his quality ceiling; the finetune's refusal-bending left the endpoint | `.rad` purged 2026-10-17 (trash-staged then emptied, Frank's approval). Re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` → `qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`, via the iq4r fetch script's hfget/verify pattern retargeted at that repo; point `MODEL_FILE` at it, redeploy. Re-**convert** is dead: the BF16 source was `rm`'d 2026-10-15 (`kld-ref`, grams, corpora and `/home/dank/logs/swift15_*.log` survive) |
+| 2026-10-05 → 10-08 | `qwen4exp-fp8-native.rad`, 177,109 MiB (173.0 GiB) — native modelopt block-FP8, no requantisation | radiance 1.0.7 line, same digest | TP=2 wht6, 200,000 × **2** slots (3 later), slab 26,071 + fp8 KV pool 3,543, host pool 40,960/rank, `--expert-vs-cache-ratio 0.82`, spec 3, MNBT 2048; that pair measured 30.08 t/s decode on the 2026-10-06 43.6k greedy probe and 4,379 t/s A-cold prefill on 2026-10-11 | native FP8 experts (56.4 GiB/rank) ate a 40 GiB pool and left KV at 3,543 MiB = 386,904 tokens, **1.93 sessions** — context economics, not quality; and the host pool could not be trimmed to pay for it (36,864 refused the boot) | rebuild-only: `scripts/download-frank-archon.sh` (official `Qwen/Qwen3.8-Flash-Next-FP8`, 131 shards, 172.8 GB, ungated) + `scripts/build-frank-archon-rad.sh` (no-recipe convert under the descriptor law); raw **and** `.rad` were `rm`'d 2026-10-15 (Frank's call, to free the volume for the iq4r download) |
+| 2026-10-05 (same day) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` safetensors, 131 shards | FreeToken PR #587 head `77a7201` (`rocm714_torch214`), image `localhost/frank-primary-freetoken:pr587-77a7201` | ONE R9700; KV sized in tokens `299,968` (a multiple of 64 — the `qsa_sparse` backend pins page 64), 2 running requests, `--moe-cache-rate 0.10`; 17.1 t/s decode | TP=2 cannot boot on ROCm: the per-rank `pynccl.cu` JIT cannot resolve `cuda_runtime.h` for gfx1201, so the second card idled | git tag `frank-archon-freetoken-pr587-20261005` (trio + `scripts/docker/frank-primary-freetoken.Dockerfile`, which exists only there); image still in rootless storage; payload needs a re-fetch, and that repo 403s until the gate is accepted on the account |
+| 2026-09-29 → 10-05 (interrupted 09-30 → 10-01 by GLM) | same orcarouter FP8 safetensors | tcclaviger vLLM fork, `docker.io/tcclaviger/vllm@sha256:da402966…` | shape G: 3 × 200,704 fp8, KV 3,166,500,000 B/rank → 401,408 tokens, arena 17.77 GiB / 7,762 slots, util 0.982, MNBT 2048, no spec, no prefetch, prefix caching ON | radiance gave real TP=2 (no RCCL) and a 2-3 min boot, and the fork never fixed its long-context class | git tag `frank-archon-vllm-shapeG-20261003` (trio + the 362-line env lore); image still in dank's rootless storage; payload needs a re-fetch. The 2026-10-01 fp8+MTP4 rung is tag `frank-archon-fp8-mtp-20261001` |
+| 2026-09-30 → 10-01 | `Intel/GLM-5.3-Flash-W4A16-AutoRound` ~170 GiB (sym int4 g128, `auto_gptq`) | same tcclaviger digest; `R4dW4a16MoEMethod` → libr4d grouped W4A16 | TP=2 + `--enable-expert-offload --expert-precise-memory`, 2 × 131,072 bf16 KV, util 0.98, MNBT 2048, 2,383 slots + 73.64 GiB pinned per rank; 9.9-10.0 t/s decode / 918 t/s prefill | the 35 t/s target is unreachable as the fork ships: 44 % of the ~93 ms/token budget is pointwise glue in an **uncompiled** GLM trunk — the expert gather is 1.05 % of device time | payload (170 GiB) + drafter (2.2 GiB) went to `~/.AGENTTRASH/20261001-glm-purge/` and were emptied; the cutover was reversed by `d91379c`, restoring the `2339fae` trio; `scripts/patches/glm53-*.patch` survive in-tree but none applies byte-exact since the 2026-10-10 rename rewrote their context lines |
+| 2026-09-27 → 09-30 | unsloth `Qwen3.8-Flash-Next-GGUF` Q8_0, 175.30 GiB / 6 shards + blockfp8 MTP4 head + mmproj; PLE derived 54,400,261,120 B | R9V radiance-vLLM fork `drwolfen/radiance-vllm-qwen4exp`, docker bundle `2dac17a2`, behind the docker→podman shim | `MAX_MODEL_LEN=131072`, 80-slot tiered cache (the policy max), MTP4, PLE **pinned**, prefix caching ON from 09-28 | the tcclaviger fork took the port for the GLM W4A16 experiment; Q8_0 had halved expert residency and a 128K envelope | host state survives: `/home/dank/r9v-state/q8xl/setup.json`, the fork checkout `/home/dank/GitPackages/radiance-vllm-qwen4exp` (local-only commits `363d606`/`30d4a32`/`2de0357`) and `.podman-shim/docker`; payload re-fetch first |
+| 2026-09-27 (same day) | `Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS`, 92.39 GiB + PLE 26.82 GiB (`34fa36f8…`) | same R9V bundle + shim | `MAX_MODEL_LEN=196608`, offload budgets 146.25 / 78.75 GiB, PLE residency `ssd` mmap, tool parser `qwen3_coder`, thinking on the `reasoning` (vLLM) channel | same-day flip back to the reference quant with the MTP4 head, at Frank's direction | `/home/dank/r9v-state/uncensored/` (`setup.json`, `restore-config-20260927.env`) is still on disk; the IQ4_XS payload was deleted in the 2026-10-17 cleanup → re-fetch, point `R9V_STATE_FILE` back, drop the two q8-only keys and the template keys |
+| 2026-09-26 → 09-27 | unsloth Q8_0, 175.30 GiB / 6 parts + shared-Q8_0 MTP 2,786,568,256 B + `mmproj-BF16.gguf` 907,542,944 B | unsloth llama.cpp b11030-mix prebuilt (commit `6ba30d05b`), ROCm gfx120X | `-ts 1,1` layer split, routed experts **and** the PLE table computed in host RAM (`-ot exps=CPU,per_layer_token_embd.weight=CPU`), 262,144 × 1 slot, then the same-day reshape to 4 × 262,144 with V `q8_0` and hot expert blocks 5/3 | the R9V radiance-vLLM fork took the port at Frank's direction (true TP=2 plus MTP4) | trio at commit `4992e5a`+; the GGUF/MTP/mmproj set is no longer on disk → re-fetch, then redeploy |
+| 2026-09-26 (same day) | `Lucebox/DeepSeek-V4.1-Flash-ROCMFP23-GGUF`, 377.2 GiB, sha256 `2daea6ca…` | `localhost/lucebox-rocm:gfx1201`, built from `Dockerfile.rocm` + 3 local patches | ONE R9700, `--max-ctx 131072`, `--chunk 8192`, q4_0 K+V, `--ds4-prefill sparse`, routed experts on CPU from the page cache | 377 GiB fits neither 63.7 GiB of VRAM nor 246 GiB of RAM → NVMe page-cache thrash; Qwen Flash-Next Q8 gave better quality, vision and 262k at comparable speed | image still in dank's rootless storage; payload gone; the quadlet and its entrypoint were deleted by `8500600` — `git show 8500600^:dankserver/frank-primary/quadlet/frank-primary.container` (and `…:dankserver/frank-primary/frank-primary-entrypoint.sh`), stop the root unit, start the user quadlet |
+
+Before 2026-09-26 the port belonged to the **`frank-medium`** service: [[frank-medium-radiance-mxfp4]]
+(vllm-radiance TP=2 MXFP4, 2026-09-20 → 09-26) is that era's page and is **not** superseded here.
+
+## Laws learned here
+
+### The radiance engine
+
+- **KV dtype law.** `--kv-cache-dtype` takes only `bf16|fp8` — ONE dtype for the whole cache,
+  so llama.cpp's exact-K / quantized-V pair (what [[frank-zealot]] runs as `--cache-type-k`/
+  `--cache-type-v`) is unservable here: the rung below bf16 is fp8 on both halves. Measured
+  per-rank cost out of the boot ledger's group table (fp8 boot 2026-10-08, bf16 boot
+  2026-10-11): `kv_attn` 13 layers / 4-token blocks pages **26.00 → 52.00 KiB**, while
+  `kv_qsa_bkey` stays **3.25 KiB at both** — the QSA block key is an index, not attention state,
+  and does not follow the flag. So 7.3125 KiB/token at fp8 vs 13.8125 at bf16: **1.89×, not
+  2×**, precisely because the bkey plane is dtype-fixed. Secondary saving: a prefix-cache entry
+  tracks the attention plane (110.50 KiB at bf16 → 58.50 at fp8, per the `kv tier transfer`
+  lines), so the same 8 GiB host tier parks ~5.8 sessions of 200k at fp8 where it parked ~2.9 at bf16.
+- **Pool sizing law.** `pool_mib ≥ tokens × 7.3125 KiB + 110 × seqs + 448` — the dtype-
+  independent terms are also in the pool: `kv_gdn_state` 54 MiB × 2 copies per sequence (+conv/
+  tail/ple ≈ 110 MiB a sequence) and the prefix checkpoint region 8 × 55.99 MiB. Deployed: 3 ×
+  200,000 at fp8 = 4,285 + 330 + 448 = **5,063 MiB**, pool 5,120, which predicts (5120 − 330 −
+  448) × 1024 / 7.3125 = **608,027 tokens** against a 600,000 contract (1.3 % slack) — every
+  MiB above 5,063 was handed to the expert slab instead. At bf16 the same three slots cost
+  8,093 + 778 = 8,871 and the slab falls to 20,249 (−9 %); four slots at bf16 cost 10,791 + 888
+  = 11,679 and the slab falls to 17,441 (−33 %); fp8 keeps 22,208 of slab at both slot counts.
+  **fp8 is not a preference, it is what funds the slot count.** Measured backings to sanity-
+  check the formula: bf16 pool 6,912 at seqs 2 backed 462,752 tokens (2026-10-11), the fp8-era
+  pool of 3,543 gave 386,904 (1.93 sessions), and the four-slot fp8 boot of 2026-10-13 19:21
+  backed 843,276. The ledger's `backs N sequences all at 200000 tokens at once` line is the
+  referee — a pair this tight is read off the boot, never assumed, and the concurrent probe is
+  its regression test. Fourth full-height conversation needs 6,601 MiB of pool: refusing it is
+  Frank's explicit call, and serving it is a `--vram-kv-mib` change, not a code change.
+- **The hard limit is per request.** The engine treats `kv group tokens >= max-model-len` as a
+  HARD limit — "one request longer than N tokens is accepted by the API and then refused as too
+  large to ever fit — this is a hard limit, not a backing ratio" — so the pool is sized for
+  `seqs × max-model-len`, not for one sequence. Radiance never truncates, at any size.
+- **One VRAM budget per card:** `weights + kv ≤ 30,969 MiB`, read off the engine's own budget
+  table (boot 2026-10-13 at pair 24000/5120): card 32,624 MiB, budgets claimed 29,120, and
+  "*3.42 GiB is neither budgeted nor accounted — the driver's context, the activation arena and
+  `--gpu-headroom-mib` come out of it; anything past those is throughput on the floor*", with
+  the card settling 2.01 GiB free against a 209.19 MiB floor (96 headroom + 113.19 kept for the
+  static region). Reserve above the engine's floor = 2,058 − 209 = 1,849, ceiling 29,120 + 1,849
+  = 30,969; the deployed pair lands 458 MiB above that floor. The 2026-10-11 bound of 29,120
+  ("31,856 − 1,710 outside both budgets − keep 1 GiB free") was a hand-picked reserve, not the
+  engine's wall. Boot with the explicit `--vram-weights-mib`/`--vram-kv-mib` pair, never
+  `--kv-cache-size-mib`; with the disk tier engaged the engine also forces ~9.6 GiB of weight
+  buffers + arena onto the weights side (~29.6 GiB/card effective).
+- **Idle KV VRAM is the expert cache's spillover — the price is paid twice.** `mover: 2275
+  slots can be lent out of the KV cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <=
+  256 tokens`; the lend grows with the pool (2.69 GiB from 3.46, 6.08 from 6.912, 4.22 today),
+  so a bigger KV pool steals weight residency from the slab AND from the lendable cache. Priors
+  on this card/model: stealing 2.14 GiB of weights to buy `MAX_MODEL_LEN 262144` cost 2.5 %
+  decode; the other direction (`--expert-vs-cache-ratio` 0.82 → 0.84, +2.5 GiB of weights) cost
+  8.9 %.
+- **The host pool moves with the slab.** `vram_weights` and `host_pool_mib` serve the per-rank
+  resident set *together*: at the fp8-KV pair (26071/3543) the ledger showed the slab full to
+  259 KiB of slack, and cutting `vram-weights` by 3.8 GiB to pay for a bf16 pool spills ~3.8
+  GiB/rank into host RAM — under-sizing the pool there is a **boot failure** (36,864 refused the
+  boot in the fp8 era). The ledger's coverage line, not the arithmetic, arbitrates.
+- **Slab → prefill: the arms disagree, the cliff does not.** 2026-10-11 exact-KV A/B (slab
+  22,208 vs the fp8-era 26,071, −14.8 %): prefill −29 %, TTFT at the full window +25 %, decode
+  inside the noise band on 2 of 3 passes, draft acceptance UP 0.661 → 0.740. 2026-10-13
+  (`probe-frank-archon-kv.py`, env header): A-cold 43.5k 22,208 → 3,397, 24,000 → 3,290, 25,600
+  → **3,584** t/s, and the 3-way 197k arm's aggregate 3,051 → 3,589 (wall 193.7 → 164.5 s), so
+  +1,600 MiB of slab bought +8.9 % single-stream / +17.6 % concurrent — read then as a SLOPE,
+  the 22,208/24,000 pair having landed inside noise of each other and falsely looked like a
+  plateau. Re-measured at 3 slots after the 2026-10-15 swap the 25,600 arm returned 3,290: the
+  3,584 did **not** reproduce, so treat 22,208-25,600 as one band until someone re-runs the
+  ladder. What is not disputed is the cliff: 26,071 → 4,379 t/s (the 2026-10-11 control at seqs
+  3 / pool 3,543), +22 % for +471 MiB — unreachable at 3 slots (26,112 + 5,120 = 31,232 vs
+  30,969) and free at 2 (pool 3,072 + slab 27,897 = the ceiling exactly). Two points inside
+  noise of each other are not a plateau, and two points are not a curve.
+- **Decode is nearly window-free; slots are paid in concurrency.** 99.0 t/s at 43.5k vs 93.3 at
+  197k (−6 % for 4.5× depth), while three streams give 113.9 / 43.3 / 10.9 = 18.4 t/s aggregate
+  — a fused-step lottery, not a fair share. Read it from leg D of `probe-frank-archon-kv.py`
+  with ≥ 128 generated tokens; an 8-token probe measures first-token noise.
+- **MNBT 2048.** 2560 collapses decode to 2.6-7.7 t/s (MTP resonance); the w4 file tier streams
+  in place only at MNBT ≤ 2048 and the activation arena scales with the chunk (32768 → 33.3
+  GiB, refused).
+- **Slots cost KV and rows, never speed.** `--max-num-seqs` carries no radiance constraint: it
+  adds 110 MiB of recurrent state per slot and declares the fused TP all-reduce over `seqs ×
+  (1 + spec)` rows, and `wht6` is bit-exact up to 25 rows (single-sequence decode and any step
+  of ≤ 25 rows is exact; only busier steps are lossy) — 3 × 3 = 9 today. Measured 2026-10-06:
+  seqs 2 and seqs 3 are statistically tied (30.9-33.1 t/s, inside the ±3 % floor). The "seqs 2
+  cannot boot" law belonged to the retired tcclaviger fork's `torch.compile` shape
+  specialization, and radiance booted seqs 2 on its first fp8-native start.
+- **One encoding per expert table** (`rad_arch_declare`): a whole expert kept bf16 is the only
+  thing legal beside the table, so a split tier (down=fp8, gate_up=w4) is unservable.
+- **Descriptor-selection law.** The plugin is picked by the exact `(arch_id, quant descriptor)`
+  pair with no fallback: a raw official FP8 checkpoint (`quant_method: "fp8"`) is refused by
+  serve *and* convert, because the shipped `qwen4exp_fp8` plugin claims `(qwen4exp, "")`. Stage
+  a `config.json` with `quantization_config` dropped — the descriptor changes, the data stays FP8.
+- **Three convert laws, each paid in a full verify+declare cycle.** Scale-name: `scale_name()`
+  appends `_scale_inv` to the *tensor* name, so stacked `…experts.gate_up_proj` keeps its scales
+  at `…gate_up_proj_scale_inv`. File-shadow: the reader opens index-named files sorted and
+  emplaces every tensor each file physically holds, first-wins — a merged PLE shard must sort
+  first (`00-ple.merged.safetensors`) or a raw shard wins the name. Symlink-through: staging
+  symlinks the raw index, so `open(path,"w")` writes through it and corrupts the checkpoint —
+  write temp + `os.replace`.
+- **Convert is card-less** (`rad-convert` declares on CPU, GPTQ solves on CPU BLAS) and
+  `--plan-only` goes first: a plan that resolves every op will boot. `rad-tune` must be
+  declared at the shapes you serve and its cache lives in the *image* (`$RADIANCE_HOME/tune`),
+  so it only survives on the `$FRANK_ARCHON_STATE_DIR/tune` bind-mount the serve script adds.
+- **Quant gates are scored at the production shape** (MNBT 2048, not the recording's 32768)
+  against the BF16 container: KL ≤ 0.03, top-1 ≥ 95 %, and the verdict is chunk-size invariant
+  (arm Δ ≈ the 0.0015 float-order band) and placement-invariant. Protected experts are
+  data-driven — energy share ≥ 3 % AND ≥ 3× routing share, even count per rank half — and the
+  PLE rides the checkpoint's own fixed scale (1.9931793212890625e-04, absmax inside the
+  2⁻¹³..2⁻¹⁰ plateau).
+
+### This host, and the deploy path
+
+- **The pinned-RAM ledger is why this is a root unit:** 2 × 44 GiB expert pools + the 47.7 GiB
+  PLE n-gram table (51B params, read from pinned host RAM over io_uring — Frank's pin law:
+  "*Native FP8. Experts from ram. Pin the PLE to ram.*") + the 8 GiB prefix host tier ≈ **147
+  GiB** charged against `RLIMIT_MEMLOCK`; dank's login session caps at 8 MiB and a user quadlet
+  is capped by the user manager, so only PID1 can grant `infinity` — and
+  `--ngram-placement ram` refuses the start when the limit is short.
+- **TTM is runtime state, and the gate that owns it raises before it refuses.** The pinned pages
+  ride `/sys/module/ttm/parameters/pages_limit` (+ `pages_max` on kernels that expose one; this
+  kernel exposes **none** — the cap is the MIN over the knobs present, and an absent knob is
+  skipped, never read as zero, because a zero would refuse a healthy cap). The kernel default is
+  **half of RAM**, so every raise dies with the reboot: the 2026-10-05 manual raise to
+  52,428,800 pages (200 GiB) was gone after the **2026-10-12 23:17 power failure**, the cap fell
+  to 125,988 MiB, and the preflight refused the start **6,919 times over ~20 h** while
+  `Restart=always` cheerfully re-ran a refusal. A manual sysfs write is therefore not an
+  acceptable repair procedure — `scripts/ttm-preflight.sh` writes `TTM_TARGET_MIB` (204,800 MiB
+  = 52,428,800 pages; pages are 4 KiB so pages = MiB × 256; the *recorded host ceiling*, because
+  writing the 150,000 floor would leave the ledger no slack) into every knob present, re-reads,
+  and only then refuses. Write failures are swallowed deliberately: the caller re-reads, and ttm
+  itself refuses a limit below the pages it has already pinned.
+- **Privilege of that knob, measured the hard way:** it is root-owned 0644 and the service runs
+  `User=dank`, where `-w` is false and a plain `echo` fails. The canonical privileged path is the
+  unit's `ExecStartPre=+…` (the `+` runs it as root even under `User=`); `sudo -n` is the
+  fallback for a human. The file is *sourced* by the serve script (which re-verifies the cap
+  before `podman run`) and *executed* by the unit. A refusal is invisible outside the journal —
+  no container is created, the unit sits in `activating (auto-restart)` on one repeating line —
+  and without the gate the load instead dies mid-pin with a bare allocator error.
+- **The prefix host tier needs real RAM:** 8 GiB of it, and with the service loaded the host
+  still has ~104 GiB available (246 GiB total) — that headroom is what the pinned ledger is
+  measured against.
+- **The deployer's blind spot is host-side unit drift.** A leftover
+  `frank-archon.service.d/bench.conf` overrode both `FRANK_ARCHON_ENV_FILE` and `ExecStart` to a
+  staging worktree and every `deploy.sh` since was green against the old configuration —
+  drop-ins are read *after* the unit's own `Environment=` lines, so they win.
+- **Never `systemctl disable` a unit on these hosts** — it removes the `/etc/systemd/system/`
+  entry, which here is the symlink into the checkout. `mask` is reversible but *fails on a
+  relinked unit*, so a bench window uses `stop` and the deploy restores the unit.
+- **The restore oracle for an experiment window is a conjunction:** `is-active` == active AND
+  `is-enabled` == enabled AND no `/run/systemd/system/<unit>` mask symlink AND the container is
+  owned by the unit and not `Stopping`. The vLLM/GLM-era engines ignored SIGTERM and answered
+  `/health` 200 for 20+ minutes while `Stopping`; a runtime mask survives `stop` and blocks
+  `start`, and a failed unit needs `reset-failed` first.
+- **`MemAvailable` is an input, not a backdrop:** the expert-offload planner reads it, so
+  anything booted beside the live service dies at plan time (`ExpertPlanError: host holds 5113
+  of 12096 swappable experts`) — and fleet law forbids GPU experiments beside production
+  anyway. Long remote work runs as a transient unit or `setsid`+disown: `nohup X &` inside `ssh`
+  races ssh teardown and can die before writing a byte.
+- **The ~40k-token corruption class was the RAM, not the engine.** Dropping DDR5 from 4800 to
+  4600 MT/s ended it, with zero `gdn_hip` faults either side; the prefix-caching / GDN re-seed
+  theory was confounded on both arms. Consumer non-ECC raises no MCE for a silent flip and a
+  60 s / 8 GiB pattern check covers 0.5 % of 246 GiB — silent RAM corruption outranks every
+  software theory.
+- **TP=2 here** is legal since the 2026-09-20 ProArt B850 swap (x8/x8 on CPU root ports,
+  `Routing+`/`ReqEn+` end to end) only because radiance's wire is `libr4d`'s own one-shot
+  all-reduce; RCCL needs atomics on the whole chain and fails `hipErrorIllegalState` without
+  them ([[Dual-R9700-Cooperation]]). `wht6` buys +5.6-6.3 % prefill for ~1.3 % of 8-stream
+  decode, `exact` is the bit-exact alternative.
+- `--prefix-cache-dir` creation is **not recursive** — the serve script `mkdir -p`s the path or
+  the disk tier silently never opens. The tier does survive a payload swap when arch and
+  tokenizer match (it recovered 900,176 blocks on the 2026-10-15 swap; `cache_n` proves a hit).
+
+### Dead ends — do not re-try
+
+- **radiance 1.2.0 as the serve pin** (`sha256:665775ed1d90de6dbb25558e07fc60d20328f1346f96f3c6e00845eb763f01fa`,
+  commit `e95d397`), benched 2026-10-06 on this exact shape with a 43.6k greedy probe: decode
+  29.21 t/s and draft acceptance 0.7123 vs the 1.0.13 control's 30.08 / 0.7549, and the control
+  reproduced the recorded 0.755 exactly — greedy acceptance is deterministic, so the drop is a
+  real numerics change (the decode loss tracks the acceptance ratio, 0.943 vs 0.945) with the
+  1.2.0 MoE weight-rotation change `92b8aaf` the named suspect. The vram-pair translation itself
+  is exact (the 1.2.0 ledger took 25.46 GiB weights / 3.46 GiB KV per card, byte-identical to
+  1.0.13 at EVC 0.82), and 1.2.0 REMOVED `--expert-vs-cache-ratio` in favour of the pair, which
+  both pins accept. Re-test 1.2.0+ once the rotation change is untangled.
+- **FreeToken TP=2 on ROCm:** the `pynccl.cu` JIT cannot resolve `cuda_runtime.h` for gfx1201 —
+  an untested edge of that PR, not a board fault.
+- **tcclaviger forks:** `--max-num-seqs 2` (`ConstraintViolationError` on `query_start_loc`);
+  `--gpu-memory-utilization 1.00` unbootable, 0.982 is the law; expert prefetch is poison (36 %
+  precision — deleting it was +40-51 % decode); TunableOp-with-offload corrupts numerics
+  (repetition loops, early EOS) — autotuning that changes text is broken tuning; ngram-only
+  drafting dies at init (`PLE inputs were not prepared`); DCP needs TP > `num_kv_heads` (2 here,
+  so 4 GPUs); 6 × 262,144 fits no KV dtype (6 sequences leave 9.18 GiB of pool where fp8 needs
+  12.47). fp8 KV on that path also removes the only MLA decode gfx12 has (the aiter op is gated
+  on CDNA > 2, so the draft step demands `…mla_decode_fwd`, which cannot exist there) and its
+  KV reserve model is dtype-blind (identical 13,600 B/token plan line) — no capacity for the breakage.
+- **DeepSeek-V4.1-Flash on 32 GiB cards:** only `--ds4-prefill exact` is legal,
+  `--ds4-fused-decode` is refused, and at ~22 % resident experts each token pulls ~187 × 12 MiB
+  from NVMe — low GPU util with a pegged link is a *capacity verdict*. Two corrections survive:
+  expert TP **does** work on gfx1201+gfx1201, and an expert-placement JSON outside the mounted
+  `/models` tree gives a silent single-tier fallback.
+- **Which experts you pin does not matter; how many does.** An IBS census of the live server
+  (`scripts/frank-archon-expert-activity.py`) measured per-block read volume flat at 0.93-1.08×
+  of mean across every CPU-resident block — top-k routing fires all of them every token, so at
+  `-ot` granularity the lever is COUNT.
+- **Never mount a diagnostic hook on `PYTHONPATH` without a marker guard:** `sitecustomize` is
+  imported by every interpreter, a child inherits the hook and spawns again — a 19-level python
+  fork bomb inside the container. Kill with `podman rm -f` (the cgroup owns every generation),
+  never pattern-kill an interpreter name on this host; use `TORCH_LOGS` for dynamo questions.
+
+## Contract (the wire)
+
+- **Live surface:** `/health` (== `/ping`), `/v1/models`, `/v1/chat/completions`,
+  `/v1/completions`, `/metrics`, `/stats`, `/sessions`, `/dashboard`. `/health` answers 200
+  JSON (model id, `in_flight`, `queue_depth`) while serving and **503 only while stopping**
+  (`core/server/admin.cpp:385`), and the listener binds *after* the full load (`main.cpp`:
+  `engine.start()` then `engine.serve()`) — during the load the port refuses connections, so a
+  bare `curl -fsS …/health` gate is honest and needs no body grep.
+- **Served id.** `FRANK_ARCHON_SERVED_NAME=frank-archon` (the wire rename of 2026-10-09 moved
+  only the alias; container, unit, scripts and env keep the service name). The env's contract
+  line is that **radiance validates the requested model name, so every client — harness, bench
+  scripts, probes — MUST send it** — while the era pages also record that a chat response
+  *echoes* the requested name, which is why `/v1/models` is the served-id oracle and a 200 chat
+  probe proves nothing about what is actually loaded.
+- **Reasoning** lands in `reasoning_content` (`--reasoning-format` defaults to `auto`; no
+  parser flags exist or are needed) and `usage` bills it separately (`reasoning_tokens`).
+  **Thinking consumes `max_tokens`**, so the classic false negative is a probe with empty
+  `content` and `finish_reason: length`/`stop` because the whole budget went inside the think
+  block: either set `chat_template_kwargs.enable_thinking=false` or budget a few hundred tokens
+  and read `reasoning_content`. Same trap, sharper, on the vision probe below.
+- **Sampler surface:** `temperature, top_p, top_k, min_p, typical_p, presence_penalty,
+  frequency_penalty, repetition_penalty, repeat_last_n, chat_template_kwargs, reasoning_effort,
+  enable_thinking, preserve_thinking` accepted; **unknown parameters are refused 400** (nothing
+  unverified can ride the wire, and an `extraBody` typo fails loudly); `max_tokens` is clamped
+  not refused; `response_format: json_object` honoured; **`logprobs` refused**; no `seed` = the
+  engine picks; `n` defaults to `max(8, max-num-seqs)`; stop strings ≤ 64 × 4096 B.
+- **Tools:** the qwen3.8 dialect from the chat template → `message.tool_calls` +
+  `finish_reason: tool_calls`. **`reasoning_effort`:** the froggeric template does not gate the
+  set (xhigh/high/medium/low all 200); the harness map is a safety net, not a requirement.
+- **Vision is native** (the checkpoint's 27-layer Qwen-VL tower through the engine's Qwen-VL
+  processor): an `image_url` content part, no legacy top-level `images` field, ≤ 4,194,304 px
+  and 16,384 patches per pass. Probe with a ≥ 256 px solid-colour PNG and a ≥ 256 token budget
+  — at 64 tokens you get empty `content` and the colour only inside `reasoning_content`.
+- **Capacity behaviour:** a prompt over `--max-model-len` is refused, never truncated; a request
+  longer than the KV pool's token count is accepted then refused as too large to fit; under the
+  3-slot contract a **fourth** full-height conversation is refused, not paged or squeezed.
+- **Observability:** `/metrics` carries vLLM-compatible names plus `radiance:*`, scraped by the
+  Dankclaw Prometheus job at `192.168.0.235:7770` over plain LAN routing (the netavark
+  published-port quirk is same-host only). The boot ledger — the referee for every shape number
+  above — is in the container log (`podman logs frank-archon`);
+  `/home/dank/logs/frank_archon.log` is where the serve script redirects `podman run`.
+
+## Files
+
+- Shape and law owner: `scripts/frank-archon.env` (measured arithmetic in its header),
+  `scripts/serve-frank-archon.sh` (gates image / `.rad` size / template / GPU nodes, retires
+  stale containers, `podman wait`), `scripts/frank-archon.service`, `scripts/ttm-preflight.sh`.
+- Payload doors: `scripts/download-frank-archon-iq4r.sh` (live), `scripts/download-frank-archon.sh`
+  + `scripts/build-frank-archon-rad.sh` + `scripts/stack-qwen38-official-fp8.py` (the fp8-native
+  rebuild), `scripts/download-frank-archon-swift15.sh` (the BF16 source a w4dn8 re-convert
+  needs), `scripts/hfget.sh` / `scripts/hf-fetch.sh`; template
+  `scripts/qwen38-froggeric-v22.5.jinja`.
+- Convert campaign (dormant, kept for a future quant): `scripts/convert-swift15-w4dn8.sh`,
+  `scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`, `scripts/run-swift15-bf16.sh`,
+  `scripts/kld-gate-swift15.sh`, `scripts/kld-then-calib.sh`, `scripts/calib-push.py`,
+  `scripts/make-calib-corpus.py`, `scripts/derive-protected-experts.py`,
+  `scripts/apply-protected-cast.sh`, `scripts/verify-swift15-ple.py`.
+- Probes: `scripts/probe-frank-archon-kv.py` (reads slots/window/dtype from the deployed env so
+  it cannot drift; leg D = decode), `scripts/frank-archon-expert-activity.py`,
+  `scripts/autoresearch-frank-archon.sh` + `scripts/autoresearch/frank-archon-measure.py`.
+- Deploy and tests: `scripts/deploy.sh` / `scripts/deploy-remote.sh` (`deploy_llm_archon`),
+  `tests/frank_archon_unit.bats` (the seams gating the KV-pool arithmetic, the VRAM pair and the
+  ttm gate), `tests/changed_tests.bats`, `tests/deployment_scoping.bats`.
+- Retired-era leftovers in-tree: `scripts/patches/glm53-*.patch`,
+  `scripts/patches/gpu_worker-kv-cap.patch` (fork archaeology; none applies byte-exact today).
+
+Related: [[frank-dragoon]] (the abliterated family), [[frank-zealot]],
+[[Dual-R9700-Cooperation]], [[deployment]], [[swift15-campaign-worklog]],
+[[dankserver-frontier-moe-feasibility]], [[x4-pair-experiment]], [[RDNA4 Container Serving]],
+[[frank-medium-radiance-mxfp4]], [[LLM/index]].

serverAppConfigs-docs/LLM/frank-dragoon-benchmark.md

@@ -1,128 +0,0 @@
----
-type: reference
-description: "This page records the benchmarks that chose the runtime for `frank-dragoon` on Dankclaw."
-tags:
-  - llm
-  - frank
-timestamp: 2026-09-03
----
-
-> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
-> See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
-> [deployment.md](deployment.md) for the active configuration.
-
-# frank-dragoon — Backend and Spec Benchmark
-
-This page records the benchmarks that chose the runtime for
-`frank-dragoon` on Dankclaw. Every number came from the live
-llama.cpp build on that machine (upstream, b10133+).
-
-## Summary
-
-- **Backend**: **Vulkan0** (2026-08-15).  At 262k ctx Vulkan decodes 48%
-  faster than ROCm (5.68 vs 3.83 t/s); ROCm only wins prefill 119 vs 106
-  t/s.  Generation is the interactive cost, so Vulkan0.
-- **Spec (2026-08-15)**: llama.cpp `--spec-default` preset (ngram-mod,
-  n_match 24 / n_min 48 / n_max 64, draftless).  Replaced DSpark, which
-  crashed the ROCm backend under two concurrent 16k prompts (see
-  [[qwen3.8-spec-matrix]]).
-- **Parallel**: 2 slots — passes the 2x32k parallel stream test.
-- **Context**: 2 slots x 262144.
-- **KV**: `q8_0` K / `q8_0` V on GPU.
-- **Batch**: 8192 / ubatch 256.
-
-> The A/B tables below are retained for history but predate the 2026-08-14
-> Qwen3.8-27B `UD-Q8_K_XL` switch.  The current production spec decision,
-> the DSpark parallel crash, and the high-context ROCm-vs-Vulkan bench are
-> recorded on [[qwen3.8-spec-matrix]].
-
-## 2026-08-14: Spec A/B (ROCm, production config, default sampling, 1.1k-token prompt)
-
-| Spec | Generation (t/s) | Parallel-safe |
-| ---- | ---------------- | -------------- |
-| DFlash (draft-dflash, n_max 8) | ~20.2 (from earlier 4k bench) | **NO** - premature EOS |
-| MTP (draft-mtp, n_max 3) | **15.3 - 19.5** | YES |
-| MTP + ngram-mod (n_max 3) | 12.0 - 13.6 | YES |
-| MTP (draft-mtp, n_max 2) | 12.4 - 14.3 | YES |
-| ngram-mod | 6.7 - 8.7 | YES |
-
-Verdict: `draft-mtp` at `n_max 3` is the winner. It nearly matches DFlash's speed, beats ngram-mod by ~2x, and does not exhibit the DFlash parallel bug. Layering ngram under MTP (12-13.6 t/s) does not pay off, the same result as layering ngram under DFlash.
-
-## Incident: DFlash Premature EOS Under Parallel Load (2026-08-14)
-
-Reproduced three times by the user. When a new task lands on a slot with high prompt-cache reuse (LCP similarity ~0.99, `f_keep` ~0.98) while another slot is generating, the generating slot is released ~1 second later with `finish_reason "stop"` and a truncated response. The client records it as a normal completion (no abort). Server log shows a `slot release` with `truncated = 0` and no `cancel task` line.
-
-Mechanism: DFlash's draft model uses one shared draft context (`ctx_dft`) across all parallel slots. When the new task's first DFlash draft pass runs after a large prompt-reuse operation, it corrupts the generating slot's draft state, and the target spuriously samples the EOG token. Known class of bug upstream (`server : do not default to multiple slots with speculative decoding`, PR #17017; DFlash multi-slot routing fixes in the beellama.cpp fork).
-
-Fix: switch away from DFlash. ngram-mod (draftless) and MTP (separate draft machinery) both pass the strict parallel test. MTP was chosen for speed.
-
-## Backend Comparison (llama-bench, raw target model, no draft)
-
-Both backends ran the same target model at 64k context in `llama-bench`
-(`-p 65536 -n 256`).
-
-| Device   | Prefill 64k (t/s) | Generation 64k (t/s) |
-| -------- | ----------------- | -------------------- |
-| ROCm0    | 220.28            | 6.99                 |
-| Vulkan0  | 72.34             | 7.00                 |
-
-ROCm prefills 3x faster. Generation is a tie in the raw benchmark.
-
-## Full-Pipeline Comparison (4k context, real served chain)
-
-These used `bench-pipeline-64k.sh` driving `llama-server` through the API at
-4k context, reading the server's own `print_timing` lines.
-
-| Spec                        | Backend | Prefill (t/s) | Generation (t/s) | Draft acceptance |
-| --------------------------- | ------- | ------------- | ---------------- | ---------------- |
-| `draft-dflash`              | ROCm    | 148.2         | **20.17**        | 0.50             |
-| `draft-dflash,ngram-mod`    | ROCm    | 149.9         | 16.93            | 0.42             |
-| `ngram-mod`                 | ROCm    | 154.0         | 8.68             | 0.57             |
-| `ngram-mod`                 | Vulkan  | 259.6         | 6.90             | 0.17             |
-
-### Interpretation
-
-- DFlash alone gives the fastest generation on ROCm (20.17 t/s).
-- Layering ngram-mod lowered ROCm generation (16.93 t/s), so it is not
-  layered in production. llama.cpp *can* layer DFlash + ngram
-  (`--spec-type draft-dflash,ngram-mod`), but it was slower here.
-- Vulkan only works without the DFlash draft. The DFlash `ctx_other`
-  setup aborts the Vulkan backend (`ggml-backend.cpp:898: pre-allocated
-  tensor ... cannot run the operation (NONE)`). Vulkan ngram generation is
-  also slowest (6.90 t/s) with poor acceptance (0.17).
-
-**Verdict: ROCm + `draft-dflash`.**
-
-## Memory Constraint at 602100 Context
-
-At `3 x 200700` context the KV cache alone is ~50 GB and the model is
-29.77 GB, leaving only ~16 GB of the ~96 GB ROCm allocation for compute
-graphs. The compute graph scales with **ubatch**, so:
-
-- batch 12288 / ubatch 4096 -> OOM (80.7 GB prefill buffer)
-- batch 6144 / ubatch 2048  -> OOM (40.9 GB prefill buffer)
-- batch 2048 / ubatch 2048  -> OOM (ubatch 2048 too large)
-- batch 2048 / ubatch 512   -> **fits and loads**
-
-The requested batch 12288 / ubatch 4096 is not feasible at this context
-size on this hardware.
-
-## Command to Reproduce
-
-Raw backend bench (ROCm shown; swap `-dev` for the other):
-
-```
-llama-bench -m <main>.gguf -p 65536 -n 256 -c 65536 -dev ROCm0 -r 1
-```
-
-Full-pipeline bench:
-
-```
-./scripts/bench-pipeline-64k.sh ROCm0 draft-dflash
-```
-
-## See Also
-
-- [[deployment]] — deploy and update process
-- [[DFlash DSpark Speculative Decoding]] — DFlash / DSpark / layering details
-- [[index]] — knowledge base index

serverAppConfigs-docs/LLM/frank-dragoon-ciru-orca.md

@@ -1,16 +0,0 @@
----
-type: reference
-title: "CIRU v4.4.1 Orca on Dankclaw (frank-dragoon)"
-description: "The retired frank-dragoon runbook (2026-09-19 to 2026-09-21, rollback target) — CIRU v4.4.1 (QSA window hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca on the Dankclaw Strix Halo iGPU: the private host build, the runtime gate, the payload, the deployed config, MTP and PLE, the client contract, and the failure modes worth knowing."
-tags:
-  - llm
-  - ciru
-  - dankclaw
-  - frank-dragoon
-  - strix-halo
-  - gfx1151
-  - qwen4exp
-  - rocm
-  - mtp
-timestamp: 2026-09-19
----

serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md

@@ -1,338 +0,0 @@
-# frank-dragoon — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 NATIVE abliterated (Dankclaw, Strix Halo iGPU)
-
-**ACTIVE since 2026-10-17.** Rollback = revert the swap commit and redeploy
-(`scripts/frank-dragoon.env`, `scripts/serve-frank-dragoon.sh`,
-`scripts/frank-dragoon.service`); the Swift1.5 payload stays on disk at
-`/home/dank/models/halogen-flashnext-v2/` (its fetch script
-`scripts/download-frank-dragoon-flashnext.sh` stays the door back), and the
-27B-era payload beside that at `/home/dank/models/halogen-qwen3.8-27b/`.
-Era write-up: [[frank-dragoon-flash-server-v2-swift15]].
-
-## Why this payload
-
-Frank's ask: serve
-`https://huggingface.co/Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated`
-with the card's recommended settings, and **save memory**.
-
-This is the **native 2026-10-11 v2 conversion** of orcarouter's pinned BF16
-Qwen3.8-Flash-Next, abliterated — the base itself, converted straight into
-the v2 `.hgn` format. It is **not** the writer transplant that carried this
-filename before: the artifact measures **66,687,678,432 B** where the
-Swift1.5 r2 measured 66,687,678,464 B — 32 bytes apart, same filename
-family, different model. Byte counts and the Hub LFS sha256 are the only way
-to tell them apart, which is why the fetch script and both gates check them.
-
-What the swap buys, measured (table below): retrieval at depth, decode,
-prefill, and **7.4 GiB of RAM handed back**. What it costs: the Swift1.5
-finetune's compressed reasoning and its small perplexity edge.
-
-## The companion law (the reason this swap needed a referee)
-
-The release publishes **only the main file**. The n-gram lookup table
-(REQUIRED — the engine reads it beside the checkpoint), the vision tower,
-and the tokenizer are peonist's, and the card does **not** establish them as
-interchangeable with this native conversion. No source-matched pair is
-published anywhere, so the only available companions are the stock ones.
-
-They are **hardlinked** out of the verified Swift1.5 payload directory
-(`scripts/download-frank-dragoon-v2abl.sh`): a hardlink is a regular file, so
-it satisfies the serve gate's no-symlink law, and it costs no disk and no
-copy next to a 62 GB payload. The engine's own checkpoint tools are the
-referee, and they answered: **NIAH 18/18** (vs 17/18 for the payload those
-files were published with), needle retrieval exact at both depths, the red-PNG
-vision probe exact, perplexity in the normal band. A wrong PLE table does not
-whisper — see Verification.
-
-## The payload — one new file, three inherited ones
-
-`/home/dank/models/halogen-v2-abliterated/`, fetched by
-`scripts/download-frank-dragoon-v2abl.sh` (`-f` + exact byte counts + LFS
-sha256 + `--verify-only`; no token on the command line):
-
-|File|Bytes|sha256|Source|
-|---|---|---|---|
-|`qwen38-flash-next-v2-abliterated.hgn`|66,687,678,432|`b5bd925f…`|Quat3rnion `@main`, fetched|
-|`qwen38-flash-next-ngram.hgn`|51,200,246,144|`3450acada…`|peonist `@5cc17cea`, **hardlinked**, REQUIRED|
-|`qwen38-flash-next-vision.hgn`|897,916,416|`d62e0ae5…`|peonist `@e053f488`, hardlinked, vision ON|
-|`tokenizer/` (6 files)| — |`tokenizer.json` LFS|peonist `@e053f488`, hardlinked|
-
-The engine stays **halogen-flash-server 0.16.2**, digest-pinned (the card
-tested 0.15.0; 0.16.2 is newer in the same line and is what the v2 format was
-qualified on in this fleet):
-
-```
-ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
-```
-
-## Shape — the card's settings, and the three deliberate deviations
-
-`HALOGEN_CTX=200704` (per-request ceiling — the card's 262,144 lowered the
-same evening, see below), `HALOGEN_KV_SLOTS=2`,
-**`HALOGEN_KV_POOL_POSITIONS=401408`** (one pool behind both slots, raised to
-2 × ctx on 2026-10-18 — see *Pool doubled for slot parity* below; the card
-recommends it equal to ctx, and that equality is where the memory is),
-`HALOGEN_SPEC_ADAPT=0`, `HALOGEN_MAX_TOK=8192`,
-`HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` (coupled pair),
-sampler `1.0 / 0.95 / 20`, `HALOGEN_WEIGHTS_LOCK=1`,
-`HALOGEN_HOST_RESERVE_GIB=20`.
-
-Deviations from the card, each on purpose:
-
-- `HALOGEN_MODEL_ID=frank-dragoon`, not the card's `halogen-v2-abliterated` —
-  the fleet naming law (service = unit = served id).
-- `HALOGEN_SPEC_ADAPT=0` — the card is silent; this is a measured 0.16.2 law
-  (the adaptive default disables the drafter per request and halves decode).
-- the sampler values are peonist's engine recipe; Quat3rnion publishes none.
-
-**The pool is one budget, not a partition.** Boot ledger form: `listening on
-127.0.0.1:8730, 2 slots over ONE <pool>-position KV pool, each request up to
-<HALOGEN_CTX>`. Both numbers were lowered to **200,704** the same evening
-(the card ships 262,144; Frank took the RAM — nothing else in the fleet asks
-for the bigger window), so `models.yml` `contextWindow` moved to **155,648**
-= ceiling − 45k with it. With pool == ctx, one maximum-length request fills
-the pool: **two long conversations can no longer share the endpoint** (two of
-≤100k can). That is acceptable here because frank-dragoon carries the `slow`
-(medium-thinking) role and the `security-reviewer` subagent, not the
-interactive workhorse (that is frank-archon).
-Frank lifted exactly that concurrency limit on 2026-10-18 and the lift is
-**deployed** (section below): the pool is again `slots × ctx`, so two
-maximum-length conversations fit the endpoint at once.
-
-**Memory ledger** (121.2 GiB unified), from the boot ledgers. The 22:20 boot
-printed `pinned 62.11 GiB in 1 range(s) in 1.6 s (41.0 GB/s)` trunk,
-`0.84 GiB` vision tower, `reserving the KV pool, 262144 positions (about
-7.2 GiB)`, and the engine's own sum: `memory: 62.9 GiB of weights locked in
-RAM (0.8 of them the vision tower's), 7.2 GiB of KV pool, …, 75.8 GiB in
-all`. The cut boot printed the same shape at the new size: `reserving the KV
-pool, 200704 positions (about 5.5 GiB)` and `… 5.5 GiB of KV pool, 5.7 GiB of
-working memory, 74.1 GiB in all`.
-The n-gram table rides as page cache — `mapped 51.2 GB,
-NOT pinned (Pin::None)` — which is exactly why every pool cut is a *double*
-saving. FLAGS.md law: **RAM the pool takes is RAM that table loses**; the ~9
-GiB the two cuts together handed back is page cache for the 47.68 GiB table
-that long prompts read.
-
-`HALOGEN_FLASH_PIN_TRUNK=0` is **rejected by the v2 loader** — the trunk
-stays pinned. Disk-backing the weights is not a lever on this format.
-
-## Pool doubled for slot parity — committed AND deployed 2026-10-18
-
-`HALOGEN_KV_POOL_POSITIONS` **200,704 → 401,408** (commit `2b12716`), Frank's
-call: frank-dragoon's total in-flight context was **half** frank-zealot's —
-the llama.cpp endpoint runs one 401,408-position pool split into two 200,704
-conversations, while dragoon ran one 200,704-position pool behind its two
-slots, so one maximum-length request filled it and any second long
-conversation had to stay under ~100k (commit message;
-`scripts/frank-dragoon.env:26-27`, `:183-186`).
-
-- **The per-request ceiling does NOT move.** `HALOGEN_CTX` stays 200,704, the
-  native wall stays 262,144 (`HALOGEN_ROPE_YARN` law), and `models.yml`
-  `contextWindow` stays **155,648** — the pool is the shared budget behind the
-  slots, not the window. The serve seam enforces `HALOGEN_CTX ≤
-  HALOGEN_KV_POOL_POSITIONS ≤ HALOGEN_KV_SLOTS × HALOGEN_CTX`
-  (`scripts/serve-frank-dragoon.sh:85-95`); 401,408 is `slots × ctx`, the top
-  of the legal band, and exactly frank-zealot's pool number — parity, not
-  approximation.
-- Measured reservation **11.0 GiB** — the boot ledger prints `reserving the KV
-  pool, 401408 positions (about 11.0 GiB)`. The preflight KV-budget line bills
-  **11.3 GiB** (`~28 KiB/position incl. block scratch + ~115 MiB/slot`) plus
-  1.3 GiB of prompt cache. The measured points beside it: 5.5 GiB at 200,704,
-  7.2 GiB at 262,144, and 14.6 GiB at the 2026-10-06 shape's 524,288
-  (`scripts/frank-dragoon.env:49-52`, `:74-76`).
-- That RAM comes out of the n-gram table's page cache — FLAGS.md law: RAM the
-  pool takes is RAM that table loses.
-
-**Deployed 2026-10-18** (`./scripts/deploy.sh dankclaw --service
-frank-dragoon`, Playwright gate 33/33). The boot ledger confirms the shape:
-`reserving the KV pool, 401408 positions (about 11.0 GiB)`,
-`2 slots over ONE 401408-position KV pool, each request up to 200704 (it
-reserves prompt + max_tokens)`, and the sum
-`62.9 GiB of weights locked in RAM (0.8 of them the vision tower's), 11.0 GiB
-of KV pool, 5.7 GiB of working memory, 79.6 GiB in all`. Prompt cache ON (12
-entries × 111 MiB at ctx 200704 = the 1.3 GiB in the KV-budget line), n-gram
-47.68 GiB still disk-backed (`Pin::None`), no clamp line. Host `MemAvailable`
-fell 26 → 22 GiB across the raise, which is the FLAGS.md collision being paid
-in page cache. **Warm reuse survived it — measured after the deploy.** One
-**101,987**-token wikitext prompt through `/v1/chat/completions` took **81.5 s**
-cold (1,251 t/s of prefill) and **0.3 s** on each of two repeats:
-`HALOGEN_CACHE_INPLACE=1` keeps the KV rows in the pool, so a warm turn prefills
-nothing and the doubling evicted nothing the cache needed. (The pre-raise
-136.7× figure was a `ttft` reading; this engine sends **no `Server-Timing`**
-header and has **no `/completion` path** — `/ping`, `/props`, `/tokenize` are
-404 too — so wall-clock around the POST plus `usage.prompt_tokens` is the only
-instrument available here.)
-
-## Measured A/B — same battery, both boots (2026-10-17)
-
-Behavior probes ran against the live service (tagged `swift15` / `v2abl`);
-the engine tools ran with the service **stopped**, per the fleet benchmarking
-law, under one identical knob set (`scripts/census-frank-dragoon.sh`).
-
-|Probe|Swift1.5 r2 (out)|native v2 (in)|
-|---|---|---|
-|`verify` on the file|—|OK|
-|NIAH 18 cases (4k/32k × 3 depths × 3 positions)|17/18 (94.4 %)|**18/18 (100 %)**|
-|wikitext-2 PPL (297,053 ids, 2 × 262,144 seqs)|**3.8772**|3.9679 (+2.3 %)|
-|KL of the new model against the old model's logits|—|mean 0.155 / median 0.057, top-1 85.1 %|
-|refusal census (5 hostile prompts)|0/5 refusing|0/5 refusing|
-|thinking tokens, mean of 6 questions (temp 1.0)|**784**|1,476 (×1.9)|
-|tool call, 1-turn / 2-turn|`{"city": "Seattle"}` / answer|`{"city": "Seattle"}` / answer|
-|vision (512 px solid red, `image_url` part)|Red|Red|
-|needle @21,568 tokens, planted 0.85 / 0.10|hit / hit|hit / hit|
-|decode, greedy, 512 tokens|46.8 t/s (spread 25.6–46.8)|50.0 t/s (steady 50.0/50.2)|
-|prefill, cold, 59,024 tokens|1,424.8 t/s|**1,604.3 t/s (+12.6 %)**|
-|prompt-cache ratio (same prompt, second pass)|136.7×|131.6×|
-|KV pool|14.6 GiB (524,288 positions)|**7.2 GiB (262,144)**|
-
-Read the KL honestly: it is a **model-vs-model** divergence, not the
-quant-vs-source comparison the tool was built for. Two different models on
-the same text *should* differ; median 0.057 with 85 % top-1 agreement is the
-signature of two sound models that disagree on content, and the worst rows
-are date/name completions. A n-gram table mismatched to the checkpoint
-corrupts the forward pass and shows up as elevated KL **everywhere** plus
-degraded retrieval — which NIAH (18/18) and the needle probes rule out.
-
-The decode line needs care: the baseline's own spread was 25.6–46.8 t/s
-because its samples were taken while the 62 GB fetch was still streaming into
-page cache, and the candidate's **first** sample after the battery's long
-prompts read 32.8 before settling at 50.0/50.2. Honest claim: **no decode
-regression, plausibly faster**; never quote either single number.
-
-## Verification — how to reproduce
-
-```
-# 1. payload: bytes + LFS sha256 + the hardlinked trio
-bash scripts/download-frank-dragoon-v2abl.sh --verify-only
-# 2. engine tools on a STOPPED service (fleet law: no GPU experiment beside
-#    a live endpoint; stop the unit first — mask fails on a relinked unit)
-bash scripts/census-frank-dragoon.sh verify    # no GPU, safe while serving
-sudo systemctl stop frank-dragoon
-bash scripts/census-frank-dragoon.sh ref       # reference dump, old payload
-bash scripts/census-frank-dragoon.sh cand      # PPL + KL/top-1 vs that dump
-bash scripts/census-frank-dragoon.sh niah-cur  # baseline retrieval
-bash scripts/census-frank-dragoon.sh niah-new  # candidate retrieval
-# 3. boot the candidate through a transient drop-in, then the battery
-bash scripts/trial-frank-dragoon.sh candidate
-python3 scripts/battery-frank-dragoon.py v2abl all
-bash scripts/trial-frank-dragoon.sh restore    # back onto the repo env
-```
-
-The battery needs two staged assets in `/tmp/halogen-eval/` (a 512 px solid
-red PNG as base64, `red512.b64`, and the wikitext-2 raw test split,
-`wikitext-2-raw-test.txt`). Laws it encodes:
-
-- **the engine bills wikitext at ~4.33 characters/token** — the harness
-  chars/4 estimate under-bills by ~8 %, and a prompt over `HALOGEN_CTX` is
-  refused with a 400 that names the exact overage (never truncated). Size
-  haystacks in characters against a **token** budget.
-- the wikitext-2 raw test split as staged joins rows with a **single**
-  newline (2,891 lines, zero blank lines), so a blank-line split returns one
-  1.29 MB "paragraph" and every budget silently becomes "everything".
-- the first request after a boot is a warmup artifact; the battery discards
-  it (`warmup` row) and never measures it.
-- `ppl`/`niah` **load the model** → stopped service. `verify` reads the file
-  back with no GPU and can run while serving.
-
-## Contract (unchanged from the Swift1.5 era)
-
-`reasoning_content` carries the thinking and counts against `max_tokens`
-(≥512-token budget on any vision probe or the endpoint looks blind while it
-is mid-thought); vision via OpenAI `image_url` content parts only, ≥256 px
-probes; `response_format` unsupported; the chat template rejects
-`reasoning_effort` outside `xhigh|medium|low` (the `models.yml` fold stays
-mandatory); unknown sampling params are a 400; the prompt cache is on by
-default, so a warm 100k follow-up costs seconds (measured after the pool raise:
-**81.5 s cold vs 0.3 s warm** at 101,987 prompt tokens). The HTTP surface is
-thin — `/health`, `/metrics`, `/v1/models` and POST `/v1/chat/completions`
-answer, while `/ping`, `/props`, `/completion` and `/tokenize` are 404 and no
-response carries `Server-Timing`: llama.cpp's timing instruments do not exist
-on this engine, so measure wall-clock plus `usage.prompt_tokens`. See
-[[frank-dragoon-flash-server-v2-swift15]] for the engine-flag detail that
-carries verbatim, and [[deployment]] for the deploy/health gate.
-
-## Production boot — the deploy ledger (2026-10-17 22:20, `fcd7630`)
-
-`./scripts/deploy.sh dankclaw --service frank-dragoon`: healthy after ~12 s,
-deploy marker `fcd7630`, e2e gate 33/33. The trial drop-in went to
-`~/.AGENTTRASH/20261017-v2abl-trial/` FIRST, so this boot is governed by the
-repo env — `systemctl show -p DropInPaths` is empty.
-
-Ledger lines that settle the shape (engine stdout, not the journal):
-
-|Line|Figure|
-|---|---|
-|preflight|`MemAvailable now 105.4 GiB` against the engine's own estimate of 62.1 weights + 6.0 working + 7.4 KV + 1.3 prompt cache|
-|weights|`62.9 GiB of weights locked in RAM (0.8 of them the vision tower's)`|
-|KV pool|`reserving the KV pool, 262144 positions (about 7.2 GiB)`; `the KV is ONE pool of 262144 positions (6656.0 MiB) carved into per-request regions`; slots `1 -> 2 POOLED, 111.0 MiB each of O(1) state`|
-|working memory|5.7 GiB at `HALOGEN_MAX_TOK 8192`|
-|total|`75.8 GiB in all` — ~25 GiB of the 121.2 GiB box left for co-tenants|
-|n-gram|`47.68 GiB from /models/qwen38-flash-next-ngram.hgn`, `checkpoint: mapped 51.2 GB, NOT pinned (Pin::None)` — disk-backed exactly as designed|
-|prompt cache|ON, resume-anywhere, 12 entries, chunk 32768, `HALOGEN_CACHE_INPLACE=1` (the KV rows are never copied)|
-|spec decode|`MTP depth-2 selectable (drafter 1, speculates while alone)`|
-|clamps|none — the requested pool is the delivered pool|
-
-Read the two KV figures together: the engine's preflight FORMULA bills 7.4 GiB
-(~28 KiB/position **including** block scratch and ~115 MiB/slot of O(1) state)
-while the RESERVATION is 7.2 GiB / 6,656 MiB. Quote the reservation for the
-RAM ledger and the formula when comparing pool sizes.
-
-Live probes on the deployed endpoint: `27*59` → `1593` with a warm thinking
-channel, tool call `{"city": "Seattle"}`, 512 px solid-red PNG → **Red** (320
-prompt tokens — the image reached the model), greedy 512-token decode 46.9 t/s
-on the FIRST request after boot, i.e. the warmup artifact; the trial's settled
-figure is 50.0. OpenWebUI's `frank-dragoon` row description re-keyed to the
-native payload (backup `webui.db.bak-20261017b`; no container restart forced —
-forcing it logs every family session out for one cosmetic string).
-
-## Ceiling lowered to 200,704 — the second production boot (`7c4dbeb`)
-
-The same evening, after the ledger above was read: `HALOGEN_CTX` and
-`HALOGEN_KV_POOL_POSITIONS` 262144 → **200704** (49 × 4096, the per-slot rung
-the llama.cpp endpoint frank-zealot already runs). The checkpoint's native
-window is 262,144 and the engine serves anything below it, so this is a RAM
-decision, not a capability limit. Nothing in the fleet asks for the bigger
-window; the harness moved WITH the ceiling (400-zone law), `models.yml`
-`contextWindow: 155648`.
-
-|Line|Figure|
-|---|---|
-|marker|`revision=7c4dbeb…`, `status=deployed`; unit `active`, `/health` 200|
-|KV pool|`reserving the KV pool, 200704 positions (about 5.5 GiB)`|
-|shape|`2 slots over ONE 200704-position KV pool, each request up to 200704 (it reserves prompt + max_tokens), max_tok 8192 … MTP depth-2 selectable (drafter 1…)`|
-|working|5.7 GiB (unchanged — it scales with `HALOGEN_MAX_TOK`, not the window)|
-|total|`62.9 GiB of weights locked … 5.5 GiB of KV pool, 5.7 GiB of working memory, 74.1 GiB in all`|
-|prompt cache|ON, resume-anywhere, 12 entries, chunk 32768, **111 MiB each at ctx 200704**|
-|clamps|none — requested pool == delivered pool|
-
-The refusal contract at the new ceiling, asked for explicitly (the 1.29 MB
-wikitext corpus through `/v1/chat/completions`):
-
-```
-HTTP 400 {"error":{"message":"the prompt does not fit: it is 297109 tokens and
-the context is 200704, 96405 tokens over. Shorten the prompt, or raise the
-context (HALOGEN_CTX, and HALOGEN_ROPE_YARN for more than the native 262144).",
-"type":"invalid_request_error"}}
-```
-
-Refused in 0.86 s — the preflight rejects before any prefill, so probing the
-ceiling costs the endpoint nothing. OpenWebUI's `frank-dragoon` row description
-re-keyed to "ONE 200,704-position KV pool at ctx 200,704" (still no container
-restart; `webui.db.bak-20261017b` holds the pre-cutover row).
-
-**What the cut cost:** with pool == ctx one maximum-length request fills the
-pool, so two 200k conversations never coexist (two of ≤100 k do), and the
-harness window is now 155,648 — the 180 k compaction target of the 262,144 era
-is unreachable because the window itself is smaller. (That pool == ctx
-concurrency limit IS superseded: the doubling deployed the next day — see
-*Pool doubled for slot parity* above. The window math stands, though: the pool
-never moves the per-request ceiling.)
-**What it bought:** 1.7
-GiB back as page cache for the 47.68 GiB n-gram table (9.1 GiB against the
-2026-10-06 shape's 524,288 pool), and a smaller prompt-cache entry.
-
-**Third production boot — the `401408` pool, deployed 2026-10-18.** Same ctx,
-same working set, same 111 MiB prompt-cache entry; the pool line reads
-`reserving the KV pool, 401408 positions (about 11.0 GiB)` and the total
-`79.6 GiB in all`. Ledger and the measured cost in
-*Pool doubled for slot parity* above; host `MemAvailable` after that boot:
-22 GiB.

serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-swift15.md

@@ -1,197 +0,0 @@
-# frank-dragoon — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, Strix Halo iGPU)
-
-**RETIRED 2026-10-06 → 2026-10-17**, superseded by the native v2 conversion:
-[[frank-dragoon-flash-server-v2-native]]. Rollback target — the payload
-stays on disk at `/home/dank/models/halogen-flashnext-v2/`,
-`scripts/download-frank-dragoon-flashnext.sh` re-verifies it, and restoring
-the era is a revert of the 2026-10-17 swap commit. The 27B payload this era
-replaced stays at `/home/dank/models/halogen-qwen3.8-27b/` with
-`scripts/download-frank-dragoon-halogen.sh`. Everything below is the era as
-it shipped; the engine-flag detail still governs the current page.
-
-## Why this shape exists
-
-Frank's ask: replace the frank-dragoon model with
-`https://huggingface.co/Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated`.
-That payload is an **abliterated Swift1.5 finetune of peonist's
-Qwen3.8-Flash-Next v2 checkpoint in the v2 `.hgn` format** — a file that
-loads ONLY in the **halogen-flash-server** engine line, NOT in the halogen
-0.1.4 container that served the 27B dense checkpoint. The model move is
-therefore a **full engine migration**, and the accepted-product facts flip
-twice in the fleet's favour:
-
-- the base is **abliterated** (the 27B halogen endpoint was a finetune of
-  the censored `Qwen/Qwen3.8-27B`; refusals were an accepted regression there),
-- **vision is back** — the v2 vision tower loads beside the checkpoint and
-  the engine runs it (the 27B halogen engine silently ignored its encoder),
-- **MTP speculative decoding returns** with the head inside the checkpoint
-  itself (the 27B shipped a DFlash2 drafter instead; comparable purpose,
-  different file plumbing),
-- the card claims **53–57 t/s greedy decode, ~1,640 t/s prefill at 34k, ~72 %
-  fewer thinking tokens** than the censored 27B (905 mean), and **262,144
-  native context**. Publisher-card numbers — verify against live traffic
-  before quoting further (see Verification).
-
-The engine is **closed-source**, ships as `ghcr.io/peonist-ai/
-halogen-flash-server:0.16.2` (the version the model card's quickstart pins),
-digest-pinned:
-
-```
-ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
-```
-
-(3.6 GB image; RepoDigest captured at pull 2026-10-06. Card-tested fallback
-if a future pin misbehaves: 0.15.1.)
-
-## The payload — a triangle of files, all sha256-verified at fetch
-
-| File | Bytes | Source repo @ revision | sha256 (LFS) |
-|---|---|---|---|
-| `qwen38-flash-next-v2-swift15-abliterated.hgn` | 66,687,678,464 | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated @ main (`69a85db`) | `9c880b8ba48deabfa06172eff1e7572e4fe465e09e97faec6e50c3f6fdcac037` |
-| `qwen38-flash-next-ngram.hgn` (REQUIRED) | 51,200,246,144 | peonist-ai/halogen-qwen3.8-flash-next @ `5cc17cea` | `3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe` |
-| `qwen38-flash-next-vision.hgn` (enabled) | 897,916,416 | peonist-ai/halogen-qwen3.8-flash-next @ `e053f488` | `d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10` |
-| `tokenizer/` (6 files, flat dir) | 22.4 MB total | peonist @ `e053f488` | `tokenizer.json` is LFS `0997f410…` (byte-identical to the 27B-era copy); the rest are plain git blobs, size-checked |
-
-Payload notes worth keeping:
-
-- The deployed checkpoint is a **compacted r2** of the card's append-only
-  original (89,775,868,032 B → 66,687,678,464 B): the abliteration workflow
-  appends and the repo's `tooling/` compact drops dead padding; live tensors
-  are byte-identical, the LFS sha256 is the contract, and the 24 GiB saving
-  is why it fits.
-- **`HALOGEN_MTP_HEAD` stays UNSET**: the MTP head is INSIDE the v2
-  checkpoint (the separate `qwen38-flash-next-mtp.hgn` sidecar is for a
-  different assembly and is deliberately not fetched).
-- The **n-gram lookup table is REQUIRED**. The engine reads it beside the
-  checkpoint (documented fallback); the card's quickstart also names it
-  explicitly via `HALOGEN_NGRAM_TABLE`, and the serve script passes both —
-  the env var is belt-and-braces over the documented beside-the-checkpoint
-  default, and both mechanisms resolve to the same file.
-- The tokenizer is fetched fresh from peonist @ `e053f488` into
-  `/home/dank/models/halogen-flashnext-v2/tokenizer/` — same flat-directory,
-  no-symlink law as every other payload (an HF cache snapshot's symlinks
-  dangle inside the container mount; both the serve gate and the deployer
-  refuse them).
-
-Fetch: `scripts/download-frank-dragoon-flashnext.sh` (`--verify-only` to
-re-check; `-f` + exact byte counts + LFS sha256 + resolve-the-CDN-then-`-C -`
-resume law throughout; no token — the repos are public).
-
-## Shape — the knobs that are not free
-
-`scripts/frank-dragoon.env` carries the full rationale inline; the table:
-
-| Env | Value | Why |
-|---|---|---|
-| `HALOGEN_CTX` | 262144 | per-slot HARD ceiling (the checkpoint's native window; a longer prompt is a 400 naming the limit, never truncation). **RAISED from 200192 on 2026-10-07 — the 400-zone law:** the harness clamps `max_tokens = min(45056, contextWindow − est − 4096)`, its chars/4 est UNDER-counts the rendered prompt by ~3 % (measured twice: est 152,938 → server 157,388, request over room by 354; est 158,906 → server 165,301, over by 2,300), and compaction only fires at 85 % of the harness window — so with window == engine ceiling, every request in the est ~146k–184k zone was a guaranteed 400. Harness window 217,088 = ceiling − 45k absorbs the error: worst-case request ≈ 219k ≤ 262,144 (≈10 % under-count headroom) |
-| `HALOGEN_KV_SLOTS` | 2 | the 27B engine's "slots>1 disables the drafter" law does NOT carry over — on this engine batching and speculation coexist (the card's quickstart ships 2 slots WITH the head). 2 slots = concurrent capacity |
-| `HALOGEN_KV_POOL_POSITIONS` | 524288 | explicit form of the default (2 × ctx), so the seam test pins the product; ~28 KiB/position ⇒ 14.0 GiB KV |
-| `HALOGEN_HOST_RESERVE_GIB` | 20 | keeps the host honest (the w4b-era lesson) |
-| `HALOGEN_SPEC_ADAPT` | **0** | MANDATORY (card): the default adaptive policy turns the MTP head off for whole requests after low acceptance and **halves decode**. The head stays on; temp-1.0 traffic pays some acceptance for it — accepted trade |
-| `HALOGEN_MAX_TOK` / `HALOGEN_MAX_TOKENS_CAP` / `HALOGEN_QUEUE_TIMEOUT` | 8192 / 65536 / 7200 | 8192 is only the per-request DEFAULT (the harness sends at most its `maxTokens` 45,056); the cap/timeout pair is COUPLED per FLAGS.md (65536 at ~10 t/s serial needs > 6550 s) |
-| `HALOGEN_TEMPERATURE` / `TOP_P` / `TOP_K` | 1.0 / 0.95 / 20 | the card's thinking recipe as SERVER-SIDE defaults — the engine HAS a sampler now, so an absent temperature is no longer greedy; models.yml extraBody matches it (consistent, not load-bearing) |
-| `HALOGEN_WEIGHTS_LOCK` | 1 | card's belt-and-braces; the engine pins weights by default on v2 (`HALOGEN_FLASH_PIN_TRUNK=0` is a w4b-only knob and is REFUSED by v2) |
-| `HALOGEN_MODEL_ID` / publish | **frank-dragoon** / 7770:8731 | served id renamed 2026-10-09 (service/unit/env keep `frank-dragoon`); engine port 8730 never published |
-| MTP depth | default 2 | FLAGS.md: 2 is best all-round, 3 for code — Frank's traffic is agent/prose |
-
-**Memory ledger** (121.2 GiB unified, the referee is the boot ledger):
-weights ~62 GiB resident + KV 14.0 + arena ~8 (`3.8 + 8.4 × 8192/16384`)
-+ vision ~2 ≈ **86 GiB engine-side**, + 20 host reserve = ~106 with
-~15 GiB margin over the box's baseline co-tenancy. The 51.2 GB n-gram
-table rides as mmap'd page cache beside the checkpoint — the card's own
-quickstart arithmetic does not bill it as resident, but WATCH the ledger
-at boot and re-size if the engine prints a clamp. Prompt cache is ON by
-default (`PROMPT_CACHE=2`): a follow-up turn at 100k ctx costs ~2 s
-instead of ~88 s cold — the big-TTFT win the 27B engine needed a manual
-`HALOGEN_CACHE_MB` pin for is built in here.
-
-## Engine contract facts (FLAGS.md of the flash-server repo)
-
-- OpenAI chat **and** `/v1/responses` **and** `/v1/messages` **and**
-  `/v1/completions`; tool calls verified on the card; `response_format`
-  still unsupported (no constrained decoding).
-- Thinking → **`reasoning_content`** (same field as every engine this
-  endpoint has run); the v2 chat template rejects `reasoning_effort`
-  outside `xhigh|medium|low` — the models.yml fold (high→xhigh) stays
-  mandatory.
-- Vision: OpenAI `image_url` content parts only (the fleet's standard
-  contract; the legacy top-level `images` field is the frank-zealot trap).
-  ≥256 px probe images or the model answers "video".
-- `HALOGEN_MAX_TOKENS_CAP` over-cap is a 400 naming the limit — never a
-  truncation. `max_tokens` INCLUDES thinking.
-- All flags read once at startup; every flag is BITWISE except
-  `HALOGEN_CK_OVERLAY`, `HALOGEN_MATMUL_TUNING_FILE`,
-  `HALOGEN_INDEXER_BUDGET`, `HALOGEN_COMPOSABLE_CONTEXT`. Unset
-  `HALOGEN_CHECKPOINT` defaults to `/models/qwen38-flash-next-v2.hgn`
-  with the lookup table beside it.
-- Prompt-lookup drafting (3,3 window) rides beside the MTP head by
-  default and is byte-identical to greedy verification where it fires.
-- Watchdog: `HALOGEN_ENGINE_WATCHDOG_S=180` with defer-logic for memory
-  stalls — on a shared host the defaults are the safe choice; the
-  restart policy stays real through the root unit's `Restart=always`.
-
-Kernel: `CONFIG_HSA_AMD_SVM=y` (CachyOS ships it); `amdgpu.noretry=-1`
-(auto) — for this IP auto resolves to retry-on, the same effective state
-the 27B engine ran on. Only revisit if the ledger shows SVM complaints.
-
-## Mechanics
-
-- **Root systemd unit** (`scripts/frank-dragoon.service`, `User=dank`,
-  `LimitMEMLOCK=infinity`, `Restart=always`, `ExecStop=podman stop -t 60
-  frank-dragoon-halogen`), driven by `scripts/serve-frank-dragoon.sh` — the
-  fleet's `podman run`-direct pattern. The script runs the container
-  FOREGROUND so the boot ledger (weights resident, pool fit, MTP head,
-  prompt cache, any clamp) lands in `/home/dank/logs/frank_dragoon.log`.
-- Serve gates mirror the deployer EXACTLY (2026-10-01 crash-loop law):
-  image digest in podman storage, byte-exact checkpoint + n-gram table +
-  vision tower, flat tokenizer with all six files as regular files,
-  GPU nodes present.
-  - **Host-RAM pin-gate preflight** (added 2026-10-06): the engine
-  refuses to pin when `MemAvailable` is below the checkpoint (62.11 GiB)
-  plus a **16 GiB floor** — pinned pages cannot be swapped, and under the
-  floor the n-gram lookup pages in from disk on every long prompt. A
-  RAM-short host crash-loops the unit (each boot dies in ~5 s at the pin
-  gate) until the RAM clears; the 2026-10-06 cutover hit this (8 boots,
-  MemAvailable 74.77 GiB vs the 78.11 GiB needed, self-resolved in ~2 min
-  when the dying 27B engine freed RAM). The serve script now checks
-  `MemAvailable` BEFORE `podman run` and WAITS up to `FRANK_DRAGOON_RAM_WAIT`
-  (default 600 s) for the RAM to clear, so a transient dip boots once
-  instead of crash-looping; if it never clears it refuses loudly. The 16
-  GiB floor is the engine's (not `HALOGEN_HOST_RESERVE_GIB=20` — a
-  separate, higher, overall-budget knob).
-- Engine stdout goes to the log, NOT the journal. The FIRST request after
-  a (re)start can sit silent ~45–60 s on this engine class (health
-  `engine_unresponsive`, 503s) and self-recovers — do not restart over it.
-
-## Verification checklist (run after any redeploy)
-
-1. Boot ledger greps in `/home/dank/logs/frank_dragoon.log`: resident
-   weights (~62 GiB), KV pool line, MTP/drafter line, prompt-cache line,
-   **no clamp named**.
-2. `curl http://192.168.0.143:7770/health` → 200.
-3. Text completion probe: reasoning lands in `reasoning_content`,
-   `timings` honest (generate ≥128 tokens; at 8 the inter_token number is
-   first-token noise).
-4. Vision probe: a 512×512 solid-red PNG through an `image_url` part →
-   "Red" (≥256 px, one word expected).
-5. Spec decode: `draft_n`/`draft_n_accepted` present in timings on
-   real-shaped traffic (SPEC_ADAPT=0 means the head is always engaged).
-
-## Client contract (harness)
-
-`~/.omp/agent/models.yml` — provider `llamacpp (dankclaw)` @
-`http://192.168.0.143:7770/v1`: `contextWindow: 217088` — deliberately
-BELOW the 262,144 engine ceiling: the 45k gap absorbs the harness's
-~3 % chars/4 est under-count (see the `HALOGEN_CTX` row above) —
-`maxTokens: 45056`, `input: [text, image]`,
-`reasoningContentField: "reasoning_content"`, the `high→xhigh` effort
-fold, `extraBody` temperature 1.0 / top_p 0.95. Compaction still rides
-frank-zealot (the `compactionModel` steering is unchanged; the input
-≈ 170k fits its 200,704 slot).
-
-## Rollback
-
-Revert the swap commit and redeploy — the 27B trio is intact on disk
-(payload at `~/models/halogen-qwen3.8-27b/`, fetch script, and the halogen
-0.1.4 image digest still in podman storage). The older ROCmFPX rollback
-door (`build-rocmfpx.sh` pin) is untouched.

serverAppConfigs-docs/LLM/frank-dragoon-gufo-qwen38-27b.md

@@ -1,350 +0,0 @@
----
-type: reference
-title: "gufo, Qwen3.8-27B UD-Q4_K_XL (DFlash2) on Dankclaw (frank-dragoon)"
-description: "The RETIRED frank-dragoon runbook (rollback target) — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL (vanilla) with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU, ONE session at the full native 262,144 window with unquantized f16 KV: the from-source build and its two host workarounds, the GGUF header evidence, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing."
-tags:
-  - llm
-  - gufo
-  - swift-1.5
-  - dankclaw
-  - frank-dragoon
-  - strix-halo
-  - gfx1151
-  - q8
-  - dflash2
-  - rocm
-timestamp: 2026-09-28
----
-
-# frank-dragoon — gufo, Qwen3.8-27B (UD-Q4_K_XL, DFlash2)
-
-> **RETIRED 2026-10-02 — the rollback target.** Superseded by
-> [[frank-dragoon-rocmfpx-q8agent-mtp]] (Q8_0_ROCMFPX_AGENT at 8.39 bpw with
-> MTP4, 17.0 t/s measured). This era was itself a return after one day of
-> [[frank-dragoon-vllm-mimo-v26-bf16]] (battery decode 6.19-9.01 t/s; every
-> ≥6-bit vLLM quant path dead on gfx1151). The shape
-> moved on the way back: payload is now **unsloth/Qwen3.8-27B-GGUF
-> UD-Q4_K_XL** (16.35 GiB, vanilla — the abliterated Swift Q8 was
-> reclaimed), and the session grid is **one session at the full native
-> 262,144 window** (`SESSIONS=1 CONTEXT_SIZE=262144`, f16 KV). The
-> Swift-1.5 Q8 history below is retained for the engine/template law,
-> which still governs.
-> **Originally ACTIVE 2026-09-28.** Replaces
-> [[frank-dragoon-unsloth-qwen38-27b-q8xl]] (stock llama.cpp, the unsloth
-> b11030 prebuilt), which had replaced
-> [[frank-dragoon-kairic-edge-qwen38-27b]]. Same host, same port (7770), same
-> alias (`frank-dragoon`).
->
-> **The payload also changed on 2026-09-28**, later the same day: the target
-> went from `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` to
-> `ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`'s
-> `UD-Q8_K_XL` — an abliterated (uncensored) fine-tune of the same base. See
-> [Payload](#payload) and, for what that costs, [the template
-> law](#the-template-law-qwen-sharp-is-gone).
-
-## Identity
-
-- **Endpoint:** `frank-dragoon` at `192.168.0.143:7770` (Dankclaw).
-- **Runtime:** [gufo](https://github.com/gufo-org/gufo) — "the Strix Halo
-  inference engine", a vertical engine built for exactly this hardware
-  (Ryzen AI MAX+ 395 / Radeon 8060S, `gfx1151`, unified memory). Pinned at
-  **v0.1.1** (commit `b0f8673`), built from source on the host to
-  `/home/dank/GitPackages/gufo-v0.1.1/`.
-- **Service shape:** root systemd unit `scripts/frank-dragoon.service`
-  (`User=dank`, `LimitMEMLOCK=infinity`), launched by
-  `scripts/serve-frank-dragoon.sh` with tunables from
-  `scripts/frank-dragoon.env`.
-
-gufo ships **no prebuilt Linux binaries** (the v0.1.0/v0.1.1 release assets
-are empty) and no Nix on this host, so the runtime is a source build. It is
-also young: v0.1.0 landed 2026-09-28, the same day as this cutover.
-
-## Why this engine
-
-gufo's own measurement of the two engines on the same Qwen3.8-27B targets
-(Strix Halo, `628ed18e` vs llama.cpp `b11069`, greedy, DFlash2):
-
-| Q8_K_XL, DFlash2 | gufo | llama.cpp | delta |
-|---|---:|---:|---:|
-| prefill, depth 0 | 457.05 t/s | 333.85 t/s | **+36.9 %** |
-| prefill, depth 131,072 | 237.24 t/s | 131.80 t/s | **+80.0 %** |
-| decode, depth 0 (mixed) | 16.08 t/s | 15.00 t/s | +7.2 % |
-| decode, depth 32,768 (mixed) | 14.41 t/s | 14.44 t/s | −0.2 % |
-| decode, depth 65,536 (mixed) | 10.87 t/s | 12.62 t/s | **−13.9 %** |
-| decode, depth 131,072 (mixed) | 8.87 t/s | 11.62 t/s | **−23.7 %** |
-| cold load to HTTP-ready | 8.17 s | 36.47 s | **4.5× faster** |
-| peak memory @ 262,144 ctx | 50.58 GiB | 48.60 GiB | +4 % |
-
-Read that honestly: **prefill, load time and short-context decode improve;
-deep-context decode on the 8-bit target gets worse.** The crossover on Q8 is
-around 32k of cached depth. frank-dragoon runs one 262,144-token conversation,
-so a long agent session spends its life on the losing side of that line
-while a short turn wins. The Q4_K_XL target does not show the regression
-(gufo is +11.7 % to +17.0 % at every depth) — that is the honest reason to
-revisit the quantization separately from the runtime.
-
-## Build
-
-`scripts/build-gufo.sh` reproduces the runtime end to end. It clones the
-pinned tag, configures the `release` preset (HIP, `--offload-arch=gfx1151`),
-builds, installs, and finishes with `gufo diagnose`. Two host facts it
-encodes, both found the hard way on 2026-09-28:
-
-- **Ubuntu's `librocwmma-dev` must not be installed.** It ships a truncated
-  rocWMMA header set (the public headers without `rocwmma/internal/`) into
-  `/usr/include`, where it shadows AMD's complete copy under
-  `/opt/rocm/include`. The first build died on `'internal/accessors.hpp'
-  file not found` from `/usr/include/rocwmma/rocwmma.hpp`, and **a `-I`
-  cannot fix it**: in HIP mode the host compilation pass searches the system
-  directories first. The package has no reverse dependencies; the script
-  removes it.
-- **`--gcc-install-dir` must point at GCC 15.** Ubuntu 26.04 has
-  `/usr/include/c++/16`, and the ROCm clang picks the newest tree, whose
-  `<atomic_wait.h>` references `std::__detail::__notify_impl` at a symbol
-  version the installed `libstdc++.so.6` (GLIBCXX_3.4.35) does not export.
-  The link then fails with `undefined symbol`. Pinning the headers to the
-  GCC the runtime library came from fixes it.
-
-Verify an installed runtime with `bash scripts/build-gufo.sh --verify`, or
-`gufo diagnose` directly — it reports `[PASS]` plus the detected
-architecture, driver and queue budget.
-
-## Shape
-
-### Payload
-
-- **Model:** `ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`,
-  file `UD-Q8_K_XL` — 27B dense, 31,457,990,272 bytes, fetched and
-  sha256-verified (`9076f99a…`, the Hub LFS oid) by
-  `scripts/download-frank-dragoon-swift.sh`. It is an **abliterated
-  (uncensored)** fine-tune of Qwen3.8-27B. The repository also publishes an
-  `mmproj-BF16.gguf`, which is **not** used: this endpoint is text-only.
-- **Licence:** the repo declares `other` — the **SWIFT OPEN LICENSE 1.0**,
-  which is *not* an OSI licence. That is a real constraint on redistribution,
-  noted here so nobody discovers it later. It is fine for a private fleet.
-- **Replaced:** `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL`, still on disk and
-  still the rollback target for the payload.
-
-The two targets are **structurally interchangeable for this engine**, which
-is why the swap needed no engine change. Header-diffed directly:
-
-| | unsloth | Swift-1.5 |
-|---|---|---|
-| `general.architecture` | `qwen35` | `qwen35` |
-| tensors (count, names, order) | 866 | 866, **identical** |
-| per-tensor `ggml_type` | Q8_0×453, F32×360, BF16×53 | **identical** |
-| per-tensor dims | — | **0 mismatches** |
-| MTP head | 4 tensors at `blk.64.nextn.*` | identical |
-| geometry | 65 blk, 5120 embd, 17408 FFN, 24/4 heads | identical |
-
-The two metadata differences that look alarming are both no-ops, and both
-were confirmed against gufo's source rather than assumed:
-
-- **`general.file_type` is 15 vs 7.** gufo's `gguf_reader.cpp` states it is
-  *"deliberately not consulted… The Unsloth UD-\*_K_XL artifacts this repo
-  targets carry a stale one — UD-Q8_K_XL records ftype 15 (Q4_K_M) … The
-  tensor table always is."* The file gufo already served lies about itself
-  here, so gufo reads the tensor table.
-- **Swift adds `qwen35.attention.recurrent_layers`.** gufo never reads that
-  key anywhere in `src/`; layer kinds come from `full_attention_interval`,
-  which both files carry as `4` and which agrees with it.
-
-### Serving shape
-
-- **Context:** one session, full 262,144 tokens (the model's native window),
-  `SESSIONS=1`. The fleet's single stream is unchanged.
-- **Spec method:** DFlash2, gufo's **adaptive** block policy
-  (`--draft-policy adaptive`, `--draft-tokens 7`). The drafter is the
-  **unchanged** z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` (1.07 GiB) — it did not
-  move with the target, which is why `DRAFTER_DIR` is a separate tunable
-  from `MODEL_DIR` and why the deployer's stray-run sweep keys on the
-  drafter's filename rather than the target's.
-- **Sampling:** greedy (`--temperature 0`) server-side, as every prior
-  frank-dragoon served. The harness overrides per request via `extraBody`
-  (1.0 / 0.95 / top-k 20).
-- **Memory:** gufo's own table puts the 8-bit target at **50.58 GiB** peak
-  with context capacity 262,144, against the iGPU's ~114 GiB GTT aperture.
-
-### Measured at the 2026-09-28 cutover
-
-Live on Dankclaw. The **target** numbers are the Swift-1.5 load test
-(a standalone run under the production flags, service stopped first); the
-**engine** numbers are from the gufo cutover earlier the same day and are
-unchanged by the payload swap.
-
-- **9.2 s** `load_started` → `load_completed` for the Swift target, against a
-  20-minute `LLM_WAIT` budget. The unsloth target loaded in 13 s.
-- **66.5 GiB** of the 116.7 GiB iGPU device at context capacity 262,144;
-  host RSS 30.6 GiB. No loader warnings, no skipped sections.
-- Contract battery, all green on the Swift target: `/v1/models` →
-  `context_length 262144`, a chat completion carrying both `content` and
-  `reasoning_content`, a `tool_calls` response with
-  `finish_reason: tool_calls`, SSE streaming, and a **two-turn tool loop**
-  (tool call → tool result → follow-up) with prior reasoning preserved.
-  Arithmetic spot-check: "17*23" → `391`.
-- Behavioural spot-check that this is the abliterated model: asked for a
-  two-sentence lockpicking scene, it wrote one without hedging or refusing.
-- From the gufo cutover, on the unsloth target: a short-prompt live request
-  returned
-  `Server-Timing: ttft;dur=549.8, inter_token;dur=24.2` (≈41 tok/s).
-  Short-prompt decode is gufo's *winning* regime; see the depth table above
-  before quoting it as the endpoint's number.
-
-### Tuning: the served flags are already optimal (2026-09-28)
-
-Frank asked for iterative decode/prefill improvement at a 32k working point
-without touching the model or the quantization. **Result: no win exists at
-this engine's served settings, and nothing was changed.** Measured on real
-repo prose and shell sources at **32,762 prompt tokens**, salted per run so
-a warm prefix cache could not help, 3–5 repetitions each.
-
-| lever | range swept | outcome |
-|---|---|---|
-| `--prefill-chunk` | 512 → 8192 | **no effect** — 410.7–412.4 t/s prefill across a 16× change |
-| `--draft-tokens` | 5 → 10 | **flat** — 13.79–15.86 t/s decode, inside run-to-run noise |
-| `--draft-policy` | fixed vs adaptive | 35.90 vs 35.50 (bench) — no effect |
-| `--min-draft-tokens` | — | **rejected by the engine**: "DFlash2 requires … 1" |
-| MTP backend | — | **unsupported for Q8**: `token_embd.weight` is Q8_0 |
-| `--context` 262144 → 40960 | diagnostic | 13.93 vs 14.71 t/s — no gain, and it would break the declared client context |
-
-The confirmation round at 5 reps **reversed** the 3-rep leaders: the pair
-that looked best (`dt9 + chunk4096`) came last at 12.86 t/s, and the
-**deployed configuration came first at 14.71 t/s**. Every difference sits
-inside the noise band.
-
-What *does* move the number is speculative decoding itself: **6.65 t/s off →
-~15 t/s on, a 5.3× step** — and that is already switched on. Prefill is
-insensitive to everything tried, which is consistent with gufo's own
-`EXPERIMENTS.md`, where dozens of kernel-level alternatives are recorded as
-rejected for "no release throughput improvement".
-
-**Do not repeat this investigation without new information.** Two traps
-cost most of the time, and both are in AGENTS.md's benchmarking policy:
-`gufo bench` reports `acceptance=0.933 ± 0.000` on an identical trace hash
-and **inverts** the draft-width ranking; and a live A/B needs ≥128 generated
-tokens or `inter_token` is first-token noise.
-
-## The template law: Qwen-Sharp is gone
-
-`scripts/qwen-sharp-v22.4.0.jinja` is the fleet's chat template — it is
-wired into `frank-archon` as `R9V_CHAT_TEMPLATE_FILE` and it was
-`frank-dragoon`'s `--chat-template-file` under llama.cpp. **frank-dragoon can no
-longer use it**, and the reason is stronger than "gufo lacks the flag":
-
-> gufo has **no Jinja engine at all.** `src/models/qwen/chat_template.cpp`
-> is 587 lines of hand-written C++ that appends strings to build ChatML.
-> There is no `--chat-template-file`, no environment override, and no
-> template body anywhere in the binary to point at. Qwen-Sharp is 453 lines
-> of Jinja2 using `namespace()`, macros, `raise_exception` and image/video
-> branches — supplying it would mean **reimplementing it in C++**, not
-> handing gufo a file.
-
-Given the choice on 2026-09-28, the decision was to **ship on gufo's stock
-Jinja and accept the loss**. What that gives up, concretely:
-
-| Qwen-Sharp feature | On gufo |
-|---|---|
-| `terse` system-prompt injection (the headline behaviour) | **gone** |
-| in-band `<\|think_off\|>` / `<\|think_xhigh\|>` control | **gone** |
-| `tool_call_format` `json` \| `xml` switch | **gone** |
-| `max_tool_arg_chars` / `max_tool_response_chars` truncation | **gone** |
-| `add_vision_id` → `Picture N:` prefixes | **gone** |
-| effort instruction strings in the system prompt | kept (gufo has its own, same text) |
-| `preserve_thinking` | kept (`--preserve-thinking on`) |
-
-The `terse` injection is the one that matters most here, and it is worth
-being blunt about the combination: this endpoint now runs an **abliterated**
-model *without* the prompt that tells it to stop waffling. If verbosity
-shows up, the cheapest fix is to send the `terse` text as an ordinary
-**system message from the client** (`models.yml` / the harness) rather than
-through a template — that recovers the behaviour without forking the
-renderer. The full-fidelity options are reverting to llama.cpp or porting
-Qwen-Sharp into the C++ renderer; both are worse trades today, and the
-second is the same class of change that produced the 2026-09-21
-"73,600-token prompt" incident.
-
-## Client contract — what actually changed
-
-Everything the fleet depended on still holds: port 7770, alias `frank-dragoon`,
-`reasoning_content`, `tool_calls`, streaming SSE, and text-only (gufo
-auto-discovers a `mmproj` beside the model; this payload has none).
-
-The differences worth knowing:
-
-- **No custom chat template — see
-  [The template law](#the-template-law-qwen-sharp-is-gone) above**, which is
-  the most consequential change on this endpoint. Thinking defaults to on at
-  `xhigh` with prior reasoning preserved (`--think auto
-  --preserve-thinking on`, spelled out in the serve script).
-- **`reasoning_effort` accepts the full OpenAI set** (`none`, `minimal`,
-  `low`, `medium`, `high`, `xhigh`, `max`), all verified 200 against the
-  live endpoint. Qwen-Sharp rejected anything outside `{xhigh, medium, low}`
-  with an HTTP 500, which is why `models.yml` still carries a
-  `reasoningEffortMap` fold for this model. **The fold is kept on
-  purpose**: removing it would silently move every `high` role from `xhigh`
-  to `high` effort. Drop it as its own change.
-- **Every field the harness sends is accepted.** `top_k`, `min_p`,
-  `presence_penalty`, `repetition_penalty` and `preserve_thinking` are all
-  outside gufo's documented request subset, and all returned 200. They are
-  applied from `models.yml`'s `extraBody` unchanged.
-- **No KV-cache quantization flag.** gufo picks its own KV dtype; there is
-  no `--cache-type-k`. The old `CACHE_RAM=8192` tunable is gone with it.
-  Prompt reuse is on by default, and `--cache-disk` (restart-safe) is opt-in
-  and currently off.
-- **Health/readiness split:** `GET /health` is process liveness and is what
-  the deployer gates on; `GET /ready` is model/backend readiness.
-  `GET /metrics` is Prometheus-format.
-
-- **Long bodies:** `--max-request-bytes` defaults to 8 MiB.
-
-## Failure modes worth knowing
-
-- **A missing runtime is a deploy error, not a restart loop.** The handler
-  gates on `LLM_DRAGOON_RUNTIME` *before* it stops anything, and the serve
-  script exits with a `build-gufo.sh` pointer rather than exec'ing.
-- **Host memory is the ceiling, and it is shared.** frank-dragoon and ComfyUI
-  both hold gfx1151 compute queues. gufo reads
-  `/sys/class/kfd/kfd/proc/*/queues/*/type` and caps itself
-  (`GPU_MAX_HW_QUEUES`, 2 resident for `serve llm`) so a full device
-  time-slices instead of failing; four HIP processes is the device ceiling.
-- **Load is seconds, not minutes** (9.2 s measured cold on the Swift
-  target), so a redeploy no longer spends 20 minutes in `LLM_WAIT`.
-- **A payload is two files in two directories.** The target and the drafter
-  no longer live together, so "the payload is present" is not one check; the
-  deployer gates on both paths and the serve script refuses to start if
-  either is absent. A symlinked "one payload directory" would hide that.
-- **A v0.1.x runtime is a moving target.** Rebuilds are not automatic —
-  `RUNTIME_DIR` names a version, and the deployer will not silently
-  substitute another.
-
-## Rollback
-
-Two independent rollbacks, because two things changed on 2026-09-28:
-
-1. **Payload only** — repoint `MODEL_FILE` in `scripts/frank-dragoon.env` at
-   `/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf`
-   and redeploy. **The 29.3 GiB target was reclaimed on 2026-09-28**, so this
-   path now needs a re-fetch first: `scripts/download-frank-dragoon.sh`, then
-   the env edit. The DFlash2 drafter was never touched and still lives in
-   that directory.
-2. **Engine** — [[frank-dragoon-unsloth-qwen38-27b-q8xl]]: the unsloth b11030
-   prebuilt is still at `/home/dank/GitPackages/unsloth-llama-b11030/`.
-   Revert `RUNTIME_DIR` and the gufo-only tunables, restore the serve
-   script's llama.cpp command, and redeploy. This is also the only way to
-   get `qwen-sharp-v22.4.0.jinja` back.
-
-# Citations
-
-- Runtime: https://github.com/gufo-org/gufo (pinned `v0.1.1`, MIT)
-- Measured tables: `docs/models/qwen3.8-27b/BENCHMARKS.md` in that repo
-- API contract: `docs/SERVER.md`; flags: `docs/CLI.md`; the C++ renderer
-  that rules out a Jinja template:
-  `src/models/qwen/chat_template.cpp` (587 lines)
-- GGUF header evidence for the payload swap: both files' headers were parsed
-  and diffed directly (architecture, all 866 tensor names in order, per-tensor
-  `ggml_type` and dims) before either was loaded
-- Build: `scripts/build-gufo.sh`; serve: `scripts/serve-frank-dragoon.sh`;
-  tunables: `scripts/frank-dragoon.env`
-- Payload: `scripts/download-frank-dragoon-swift.sh` (target),
-  `scripts/download-frank-dragoon.sh` (DFlash2 drafter)
-- Predecessor: [[frank-dragoon-unsloth-qwen38-27b-q8xl]]

serverAppConfigs-docs/LLM/frank-dragoon-halogen27b-dflash2.md

@@ -1,200 +0,0 @@
-# frank-dragoon — halogen container, Qwen3.8-27B + DFlash2 (Dankclaw, ACTIVE 2026-10-05)
-
-frank-dragoon is the Strix Halo iGPU endpoint on Dankclaw (`:7770`). Since
-2026-10-05 it runs the **halogen** engine — `ghcr.io/peonist-ai/halogen` at a
-pinned digest, launched by `scripts/serve-frank-dragoon.sh` under a root
-systemd unit as a podman container — serving
-**`peonist-ai/halogen-qwen3.8-27b`** (`qwen3.8-27b-p1w4d-d2.hgn`,
-35,865,565,184 B ≈ 33.4 GiB, sha256 `274c3fc7…`, 1352 tensors) with the
-**DFlash2** drafter at **one slot x 200,700 ctx**.
-
-It replaced the ROCmFPX llama.cpp fork endpoint ([[frank-dragoon-rocmfpx-q6agent-uncensored-mtp]]),
-which remains the rollback target with its payload on disk.
-
-## Why swap
-
-|Metric|ROCmFPX Q6_0_ROCMFPX_AGENT + MTP (retired)|halogen qwen3.8-27b + DFlash2 (active)|
-|---|---|---|
-|Prose decode|27.6–28.1 t/s measured (4.8k depth, acceptance 0.868)|31.7 t/s mean across eight text workloads (publisher card, bf16 reference hardware)|
-|Copy-echo decode|31.4 t/s|publisher reports echo inside the same 31.7 mean|
-|Context|1 x 262,144|1 x 200,700|
-|Format|GGUF (fork-private tensor types 102/103)|`.hgn` — loads in NO other engine|
-
-The swap was accepted with eyes open on two product regressions, both
-confirmed by Frank before deploy:
-
-1. **Text-only.** halogen loads the language tower only; README "honest
-   limits": *"The vision encoder is ignored; image inputs are not supported."*
-   The mmproj-era vision role moves wholly to frank-zealot.
-2. **Censored base.** The checkpoint is a finetune of the censored
-   `Qwen/Qwen3.8-27B` base, built to make its DFlash2 head strong. The
-   ROCmFPX AGENT file was an *Uncensored* build; refusals can return.
-
-## Shape and why
-
-`scripts/frank-dragoon.env` carries every tunable with its evidence; the load-
-bearing numbers:
-
-- **`HALOGEN_KV_SLOTS=1` + `HALOGEN_DRAFTER=2`** — the engine offers batch OR
-  speculation, never both (FLAGS.md: "Batching and speculation are exclusive:
-  a step batches when more than one conversation is running, so the drafter
-  stays off with more than one slot"). Slots>1 silently loses DFlash2; the
-  sibling halogen engine measured 41 -> 14 t/s when slots went up. frank-dragoon
-  was a one-conversation endpoint before (PARALLEL=1), so the contract holds.
-- **`HALOGEN_SLOT_CTX=200700`** — the per-conversation HARD ceiling: a longer
-  prompt is a 400 naming the limit, never a truncation. The harness side is
-  wired to it: `models.yml` `contextWindow: 200700`, and OMP compacts at 70%
-  = 140,490, leaving 60,210 of the slot for the reply.
-- **`HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200`** — coupled
-  pair (FLAGS.md table): a request at the cap must not outlive the timeout.
-  65536 > the harness's 57,344 `maxTokens` so an eager request never meets
-  the engine's 400.
-- **KV math:** 1 slot x 200,700 x 64 KiB/token (d2 KV = 2x fp8) = 12.2 GiB
-  pool, inside the ~65 GiB left after 33.4 GiB mmap'd weights + engine working
-  set + host baseline on the 121.2 GiB box. The engine sizes the pool itself
-  and **prints a boot ledger naming any clamp** — the post-deploy check reads
-  that ledger, not a formula.
-- **`HALOGEN_CACHE_MB=16384` (pinned 2026-10-06)** — auto is OFF on this
-  box: the floor is 2 x a full-context entry (36.9 GB at 262k; one entry
-  is ~13.7 GB at our 200,700-slot x ~70 KB/token) and the ledger had
-  35.7 GB free, so the engine refused and every deep agent request
-  re-prefilled its whole 79-81k context — 84.9-179.8 s measured
-  (serve_api lines, 2026-10-06) for ~500-650 new tokens per turn. The
-  README prices the cache at ~20x time-to-first-token on a warm turn;
-  entries evict LRU by bytes; the reserve (HALOGEN_CACHE_RESERVE_MB,
-  default 8192) is re-checked against live MemAvailable on every store.
-- **No server-side sampler:** an absent temperature is GREEDY. The card's
-  thinking recipe (temp 1.0 / top_p 0.95) lives in `models.yml` `extraBody`
-  — on the wire only. The template defaults `reasoning_effort` to **xhigh**
-  when a request sends none, and **thinking tokens consume the
-  `max_tokens` budget**: a budget exhausted mid-thought returns
-  `finish_reason: "length"` with EMPTY `content` and the partial reasoning
-  in `reasoning_content` — clients must read both fields.
-
-## Client contract deltas vs llama.cpp
-
-|Behavior|halogen|
-|---|---|
-|Prompt over slot|400 naming the limit (no truncation)|
-|`max_tokens` over cap|400 naming the limit (never silent truncation)|
-|full-length request outliving `HALOGEN_QUEUE_TIMEOUT`|503s for everyone queued|
-|`timings`|every response carries llama-server-shape `timings` (`draft_n`, `draft_n_accepted`) — the drafter check reads the RESPONSE, and it proves speculation fired|
-|`response_format`|NOT supported (no constrained decoding)|
-|thinking field|`reasoning_content` (same as before)|
-|`reasoning_effort`|same Qwen3.8 template; outside {xhigh,medium,low} rejected — models.yml fold stays; unset defaults to xhigh, `none` disables thinking|
-|tools|server-side tool calls advertised — round-trip MUST be re-probed after deploy|
-|image input|ignored/error — route to frank-zealot|
-
-## Deploy mechanics
-
-- **Image:** `ghcr.io/peonist-ai/halogen@sha256:dc0a39a0016d6cfc58a197978febaafdf8d28403f724ded6111d98b5fb7ac0ea`
-  (published tag 0.1.4). The deployer gates the exact digest exists in podman
-  storage before dispatch. NOT the 2026-09 `halogen-flash-server` repo — that
-  was the Flash-Next w4b era, different image, retired.
-- **Payload:** `scripts/download-frank-dragoon-halogen.sh` (fetch law: `curl -f`,
-  exact byte count, LFS sha256 where the Hub publishes one, `--verify-only`,
-  token never in argv) into `/home/dank/models/halogen-qwen3.8-27b/`. The
-  tokenizer is a FLAT DIRECTORY (`tokenizer/`, 5 files — tokenizer.json,
-  vocab.json, merges.txt, chat_template.jinja, tokenizer_config.json),
-  mounted whole at `/tokenizer`; the engine's `HALOGEN_TOKENIZER` default is
-  that directory, so the container gets no tokenizer flag (model-card
-  docker-run parity). The deployer and serve script refuse a missing file or
-  ANY symlink: an HF cache snapshot symlinks each entry into `blobs/` and
-  those links dangle inside the container mount.
-- **Unit:** `scripts/frank-dragoon.service` (root, `User=dank`, relinked into
-  `/etc/systemd/system/`), `ExecStart` = serve script, `ExecStop` = `podman
-  stop -t 120 frank-dragoon-halogen`, script runs `podman run` FOREGROUND
-  (2026-10-06: the 2026-10-05 `-d` + `exec podman wait` shape captured
-  only the container ID — the engine ledger existed nowhere on disk and
-  the post-deploy grep below read a one-line hex file; foreground run
-  returns the container's exit code so `Restart=always` is real and the
-  ledger lands in the log). Engine stdout goes to
-  `/home/dank/logs/frank_dragoon.log` (the boot ledger lives there, not in
-  the journal).
-- **Cutover:** `deploy_llm_dragoon` in `scripts/deploy-remote.sh` sweeps the
-  legacy bare `llama-server --model /home/dank/models/kingjones-*` process and
-  the legacy `frank-dragoon` container name by NAME (never `pkill -f
-  llama-server` — that hits frank-zealot), retires the old quadlet if the 2026-09
-  halogen era's unit lingers, waits port 7770 free, relinks, health-gates.
-- **Bench law:** stop+mask the unit before any GPU experiment on this box
-  (AGENTS.md); `/health` answers 503 while the 33.4 GiB payload loads — that
-  is loading, not failure.
-
-## Verification (post-deploy checklist)
-
-1. `grep -iE 'clamp|kv|pool|drafter|dflash|cache' /home/dank/logs/frank_dragoon.log`
-   — ledger shows the pool at the asked shape, DFlash2 active, the
-   prompt cache line naming the pinned budget, NO clamp line. EXPECT the
-   WARNING "smaller than ONE full-context entry (18.4 GB at
-   kMaxCtx=262144)": it sizes against the NATIVE window, not our
-   200,700 slot — the deepest possible entry (~14 GB at ~70 KB/token)
-   fits the 16 GB pin, and the 262k entry it fears cannot exist (over
-   slot is a 400). Probe-proven 2026-10-06: the warm turn hit with
-   cache_n=16,384.
-2. `curl -s http://192.168.0.143:7770/health` -> 200.
-3. Real completion with `reasoning_effort` + temp/top_p from the wire recipe;
-   `reasoning_content` populated; decode rate from the timings.
-4. Spec decode fires: multi-token step behavior visible in rate; compare
-   against DRAFTER=0 arm if in doubt (never against gufo — bench law).
-5. Harness round-trip: an OMP call through `llamacpp (dankclaw)/frank-dragoon`
-   incl. a tool call (V1TOOLS).
-6. Prompt > 200,700 tokens returns the 400 naming the limit (contract test).
-
-## Tuning notes (2026-10-06)
-
-Measured on real agent traffic (serve_api lines) the day after cutover:
-
-- **Decode is depth-bound, and the KV path is the reason.** Shallow
-  prompts (< 1k tokens) decode 30-50 t/s at commit 4-6.3/round; 46k runs
-  5.5, 63-67k runs 4.1-6.5, 79-81k runs 3.3-3.8. The round-time slope is
-  ~8 us/token of context: the KV pool reads at ~8 GB/s effective while
-  the weights stream ~180-240 GB/s (the engine's README puts batch-1
-  weight streaming at the 240 GB/s hardware wall). Commit also falls
-  from 4-6 to 2.5-3 accepted/round as the conversation grows.
-- **No supported flag touches decode.** `docs/FLAGS.md`: HALOGEN_W4A4 /
-  W4A4_EXCL are prefill-only ("decode is unaffected either way"); the
-  undocumented kernel levers are declared unsupported and their defaults
-  are the measured winners. 0.1.4 is the latest release (CHANGELOG
-  head; kernels unchanged since 0.1.0). The publisher's tg bench never
-  measured depth — its tg rows run over the ten SHORT eval prompt
-  shapes — so the README's "decode barely degrades at depth" claim was
-  never exercised at 46-80k.
-- **A/B'd the one FA lever with public precedent (2026-10-06 stop-window
-  probe): rejected.** Issue #12's `HALOGEN_FA_TQ=256` (the 64 GB-carve
-  crash workaround) vs the default tile — same 80k filler prompt,
-  128-token essays, the runtime's own `timings`: decode 6.35 vs 6.34 t/s
-  cold and 5.74 vs 5.76 warm (±0.3 % = noise; draft acceptance identical,
-  53/74 and 58/69), while prefill LOST 9.5 % on this topology (447.5 ->
-  404.8 t/s). The depth collapse is not an FA tile artifact — it sits
-  deeper than the knob. The engine binary advertises 40+ undocumented
-  `HALOGEN_*` kernel levers (FA/DN/MATMUL families, readable via
-  `grep -aoE "HALOGEN_[A-Z0-9_]+"` on the binary); `FA_TQ` is the only
-  one with public evidence, and it moved nothing that matters here.
-- **Routing is the applied mitigation (2026-10-06):** the harness's
-  deep-conversation roles (`slow`, `advisor` in `~/.omp/agent/config.yml`)
-  moved to frank-archon (radiance on Dankserver: fp8 KV at
-  7.32 KiB/token = 8.7x less KV to read per round at depth, MTP depth 3,
-  200k window, prefix-cache host+disk tiers). frank-dragoon keeps OpenWebUI
-  and shallow bursts, where it runs 30-50 t/s. frank-archon's
-  depth-decode rate is designed-for but not yet measured on record.
-- **The lever that existed was the prompt cache, and it was OFF** (see
-  the shape bullet). The pin turns the 85-180 s re-prefill per deep
-  turn into a resume; decode itself is expected UNCHANGED by the pin —
-  the cache resumes prefill, it does not touch the round loop.
-  Probe-proven (two synthetic 17.3k turns, 2026-10-06): cold prefill
-  27.0 s (640 t/s, cache_n 0) -> warm turn cache_n 16,384, prompt_n 946
-  (the extension + one 2048-aligned chunk), prefill 1.7 s — 16x on the
-  pair, answer correct warm.
-- **Everything else stays put.** dflash2 over mtp (publisher 31.71 vs
-  26.88 mean; at depth both pay the same KV term and dflash2's higher
-  commit wins), KV_SLOTS=1 (batch XOR speculation), SLOT_CTX=200700
-  (Frank's window), W4A4 shipped (+9 % prefill, decode unharmed).
-
-## Rollback
-
-Revert the frank-dragoon swap commit and redeploy `dankclaw --service frank-dragoon`:
-the ROCmFPX payload (`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/`),
-the fork build (`/home/dank/GitPackages/kingjones-ROCmFPX/build`), and its
-`scripts/build-rocmfpx.sh` pin (commit `36e9acd`, seam-tested) all remain on
-disk. The halogen payload can stay: it costs nothing while the unit points
-elsewhere.

serverAppConfigs-docs/LLM/frank-dragoon-ik-uncensored.md

@@ -1,149 +0,0 @@
----
-type: reference
-description: frank-dragoon on Dankclaw serving the mradermacher Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M GGUF via ik_llama.cpp on the RTX 3090 + Strix Halo hybrid — measured performance envelope, root-cause analysis, tuning levers, and traps
-tags: [llm, frank-dragoon, ik_llama, dankclaw, rtx3090, strix-halo, performance]
-timestamp: 2026-09-16T03:30:00-07:00
----
-
-# frank-dragoon — ik_llama.cpp, Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M
-
-**RETIRED 2026-09-16, after one day.** Replaced by [[halogen-flash-server]],
-which was the runtime before it as well — the fleet went ik → halogen → back.
-Its unit, env, serve script and the Qwen-Sharp template are archived in the repo
-under `scripts/.AGENTTRASH/*.ik-llama.cpp` and `*.ik-llama-qwen-sharp`, and the
-payload, build and chat template stay on disk. The measurements below are not a
-stale plan: they are the complete account of what this model costs on this
-hardware, and they are why the fleet went back. It served
-`mradermacher/Qwen3.8-Flash-Next-Uncensored` (111 GiB GGUF + mmproj-f16) at
-`192.168.0.143:7770` while it ran.
-
-## Shape
-
-- Runtime: ik_llama.cpp master `d85a1d7`, built at
-  `/home/dank/GitPackages/ik_llama.cpp/build` with `-DGGML_CUDA=ON
-  -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_AVX512=ON -DGGML_AVX512_VBMI=ON
-  -DGGML_AVX512_VNNI=ON -DGGML_AVX512_BF16=ON -DLLAMA_CURL=OFF`.
-- Unit: **root systemd unit** `/etc/systemd/system/frank-dragoon.service`
-  symlinked to `scripts/frank-dragoon.service` in the repo; enabled at boot;
-  `Restart=always`. Serve script `scripts/serve-frank-dragoon.sh` + env
-  `scripts/frank-dragoon.env`.
-- Topology: 3090 (PCIe x4 in the M.2 dock, link downgraded from x16) carries
-  dense/attention/KV (~11.5 GiB); ALL MoE expert tensors (~84 GiB) and the
-  ~33 GiB PLE n-gram table stay in LPDDR via CPU buffers.
-- Client contract: greedy default (`--temp 0.0`), reasoning on (deepseek
-  format, unlimited budget), vision via mmproj, slot-state persistence under
-  `scripts/slot-state`, 262144 ctx, 1 slot.
-
-## Measured performance envelope (2026-09-15/16 campaign, all single-instance verified)
-
-| config | decode tok/s | prefill tok/s | verdict |
-|---|---|---|---|
-| shipped (T16, `-ot exps=CPU`, NO_PINNED_WEIGHTS) | **3.8-4.0** | ~193 at 1.8k / 191 at 32k | shipped |
-| T8 / T4 / T2 / T1 | 4.18 / 4.20 / 4.26 / 3.97 | small-prompt only | decode flat across threads |
-| `-ncmoe 42` (6 layers' experts on GPU) | 4.54 | 32 (small prompt) | VRAM 22.0/24.6 GiB — fragile |
-| `-ncmoe 48 -thp` | 4.60 | 18 | **110 GiB anon host buffer — co-tenant killer, never ship** |
-| `--prefetch-experts` | 4.03 | — | no effect (page cache already resident) |
-| `--defer-experts --defer-ple -thp` | crashes | — | experts under `-ngl 999` still target CUDA0 → cudaMalloc OOM |
-| mainline llama.cpp `37b53fd` (`--n-cpu-moe 48`) | **1.01** | 34 | ruled out — CPU MoE kernels 4x slower than ik's iqk |
-
-- Large-prompt prefill ≈ 192 tok/s is the **CPU AVX512-VNNI compute ceiling**
-  for CPU-resident experts (expert weight reads amortize across the batch;
-  compute-bound). Halogen's "4x llama.cpp family" prefill claim is therefore
-  unreachable with experts on CPU — the physics caps hybrid MoE prefill here.
-- Decode is gated by a **serial per-token critical path**: ~100 CPU↔GPU
-  backend splits per token (`graph splits = 100` in the log); the 3090 sits at
-  **5% SM utilization** during decode; thread count does not matter (T1 ≈
-  T16 ≈ 4 tok/s). The GPU sections are launch/sync-bound, not compute-bound.
-
-## Root cause chain (perf evidence)
-
-1. `perf` at T16: ~53% cycles in `ggml_graph_compute_thread` + ~37% in
-   `ggml_compute_forward_mul_mat_id`, of which **~98% is the `pause`
-   instruction** (thread-pool barrier spin). Actual vec_dot math: ~4%.
-2. T1 = T16 decode (3.97 vs 3.8-4.0) ⇒ the barrier spin is threads waiting on
-   a serial critical path, not parallel overhead.
-3. GPU 95% idle during decode ⇒ the critical path is the GPU-section
-   launch/sync overhead per split (x4 dock link, eager kernel launches).
-4. ik's CUDA-graph capture would collapse those launches but **aborts** on
-   qwen4exp: the per-layer hyper-connection broadcasts (`ggml_repeat_4d`,
-   4 sites in `src/graphs/build_qwen4exp.cpp`) resolve to a REPEAT node whose
-   CUDA path does a capture-illegal operation
-   (`CUDA error: operation not permitted when stream is capturing` → abort).
-5. Memory placement is NOT the lever: mmap-off/anon/THP/deferred variants all
-   land at 3.8-4.6 tok/s.
-
-## Why the 30 tok/s halogen bar is out of reach (honest record)
-
-Halogen's engine held the whole 115.55 GiB w4b checkpoint resident in LPDDR
-(`HALOGEN_FLASH_PIN_TRUNK=1`) and served from one engine. This deployment must
-split 84 GiB of expert tensors across a 24 GB GPU + CPU, and every public
-runtime pays the split penalty above. Paths to >30 tok/s, in order of
-realism: (a) a runtime with capture-safe CUDA graphs + dedicated hc ops —
-mainline is building exactly this (ggml-org PR #28901 "qwen4exp: add hc ops");
-re-evaluate mainline when its hc-ops + graph handling mature; (b) experts on
-GPU — impossible, 84 GiB > 24 GiB; (c) a smaller model.
-
-## Upstream issues tracking the slow TG path
-
-ik_llama.cpp: #2374 (QSA opt POC), #2406 (single-threaded sigmoid), #2433
-(cache-miss full reprocess), #2460 (PLE history correctness). MTP/speculative
-decoding is unavailable for this GGUF — mradermacher stripped the NextN head
-(verified: no `blk.48.*`/nextn tensors in the file).
-
-## Traps (learned the hard way 2026-09-15/16)
-
-- **Pinned host buffers strand the host**: a default-policy start
-  page-locks ~108 GiB (`CUDA_Host buffer`), evicting co-tenants into swap and
-  starving sshd until a power cycle. `GGML_CUDA_NO_PINNED_WEIGHTS=1` + the
-  serve-script guard exist for this. `-ncmoe N` with default NO_PINNED
-  settings produces the same 108 GiB pinned buffer — never run it without the
-  guard.
-- **Any `-ot`/`-ncmoe` override disables mmap** for the whole model load
-  (`create_tensors` returns false → `ml.use_mmap = false`): the "mmap-backed
-  CPU buffer" comment in the serve script is wrong on the mmap word; the
-  buffer is anonymous. All placement variants perform alike anyway.
-- **`--defer-experts` does not keep experts off the GPU** under
-  `-ngl 999` — only PLE defers (32.78 GiB). With ngl 999 the load dies with
-  `cudaMalloc failed: out of memory` at ~79.7 GiB on device 0.
-- **The systemd unit self-heals through manual tests**: `pkill` the server and
-  the unit respawns it ~4 s later (Restart=always) — two instances, port
-  races, phantom measurements. For manual experiments:
-  `sudo systemctl stop frank-dragoon.service` first (it is a root unit; plain
-  `systemctl --user` lines in older docs are stale), and restore with
-  `sudo systemctl start frank-dragoon.service`.
-- Verify "HTTP server listening" in the CURRENT log and
-  `pgrep -xc llama-server` == 1 before trusting any measurement.
-- FA must stay ON — FA-off crashes the loader on this arch.
-- Do not run builds concurrently with decode measurements (`-j16` starves the
-  service and corrupts both results).
-
-## Rollback
-
-Stop and disable the root unit, trash it, restore the halogen quadlet from
-`~/.AGENTTRASH/20260915-ik-swap/halogen-quadlet` to
-`~/.config/containers/systemd/`, `systemctl --user daemon-reload`, start;
-the .hgn checkpoint and the kingjones STRIX llama.cpp payload remain on disk.
-
-## Hybrid campaign envelope (2026-09-16, 3090 + Strix Halo)
-
-Why this deployment could not be tuned into a production endpoint: decode
-3.8–4.3 tok/s flat across thread counts (T1 ≈ T16); prefill ~192 tok/s at
-1.8k–32k prompts = the **CPU AVX512-VNNI compute ceiling** for CPU experts.
-The decode root cause is structural: ~100 CPU↔GPU backend splits per token
-with the GPU at 5 % SM during decode — launch/sync-bound over the PCIe x4
-dock, not compute- or memory-bound. T16 profiling shows ~90 % of cycles in
-`pause` (ggml_barrier spin): threads wait on the serial critical path.
-Memory placement (mmap/anon/THP/deferred) does not matter — all variants
-landed 3.8–4.6.
-
-- CUDA-graph capture aborts: qwen4exp's hyper-connection broadcasts use
-  per-layer `ggml_repeat_4d`, whose CUDA path is capture-illegal; until ik
-  gains capture-safe repeat/hc ops (mainline PR #28901 direction), graphs
-  stay dead.
-- Mainline llama.cpp itself runs 1.0 tok/s on this model (CPU MoE kernels
-  4× slower than ik's iqk) — do not switch engines for speed.
-- MTP/speculative decoding is impossible for mradermacher i1 GGUFs: the
-  NextN head is stripped (no `blk.48` tensors).
-- Tuning lever ranked first: `-ncmoe 42` (6 layers' experts on the 3090) →
-  decode 4.54, small-prompt prefill 32 — but VRAM 22/24.6 GiB at 262k ctx,
-  an operator OOM-margin decision, and it needs `GGML_CUDA_NO_PINNED_WEIGHTS=1`.

serverAppConfigs-docs/LLM/frank-dragoon-kairic-edge-qwen38-27b.md

@@ -1,48 +0,0 @@
----
-type: reference
-title: "Kairic Edge Qwen3.8-27B IU4 (W4A4) on Dankclaw (frank-dragoon)"
-description: "The RETIRED frank-dragoon runbook (2026-09-21, same-day swap; replaced by [[frank-dragoon-unsloth-qwen38-27b-q8xl]]) — the Kairic Edge runtime (ciru-ai/ROCmFPX, branch kairic-edge-qwen38-27b-v1.2, commit 205a3e5, TheRock 7.15.0, Composable Kernel) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (27B dense, IU4 W4A4) on the Dankclaw Strix Halo iGPU: the source-built runtime, the full-window/one-stream shape, the MTP depth 4 + n-gram spec method, the three PromptForge sidecars, the compatibility mode, the payload, the client contract, and the failure modes worth knowing."
-tags:
-  - llm
-  - kairic-edge
-  - rocmfpx
-  - dankclaw
-  - frank-dragoon
-  - strix-halo
-  - gfx1151
-  - iu4
-  - w4a4
-  - rocm
-  - mtp
-  - promptforge
-timestamp: 2026-09-21
----
-
-# frank-dragoon — Kairic Edge Qwen3.8-27B IU4 (W4A4) (RETIRED)
-
-> **RETIRED 2026-09-21 (same-day swap).** This was the ACTIVE frank-dragoon
-> runbook on 2026-09-21 before it was replaced by
-> [[frank-dragoon-unsloth-qwen38-27b-q8xl]] later the same day. The payload
-> and runtime remain on disk as the rollback target.
-
-## What it served
-
-- **Runtime:** Kairic Edge (ciru-ai/ROCmFPX, branch
-  `kairic-edge-qwen38-27b-v1.2`, commit 205a3e5, TheRock 7.15.0,
-  Composable Kernel), root systemd unit, port 7770.
-- **Model:** `jcbtc/Qwen3.8-27B-IU4-Kairic-Edge` (a 27B dense model, IU4
-  W4A4) — main GGUF + FFN/GDN/GDN-Output `.pfs` sidecars.
-- **Shape:** one 262,144-token conversation (single stream), native MTP
-  depth 4 + n-gram drafting, compatibility mode.
-- **Memory:** the 114 GiB GTT aperture on the Strix Halo iGPU.
-
-## Rollback
-
-Payload and runtime intact on disk. See [[frank-dragoon-unsloth-qwen38-27b-q8xl]]
-for the current ACTIVE runbook and [[frank-dragoon-unsloth-qwen38-flash-next]]
-for the predecessor.
-
-# Citations
-
-- Kairic Edge runtime: https://github.com/ciru-ai/ROCmFPX
-- Model: https://huggingface.co/jcbtc/Qwen3.8-27B-IU4-Kairic-Edge

serverAppConfigs-docs/LLM/frank-dragoon-rocmfpx-q6agent-uncensored-mtp.md

@@ -1,177 +0,0 @@
----
-type: reference
-title: "frank-dragoon on Dankclaw — ROCmFPX llama.cpp, Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP"
-description: "The active frank-dragoon runbook — the kingjones30/ROCmFPX llama.cpp fork (pinned commit, gfx1151) serving kingjones777's Uncensored Q6_0_ROCMFPX_AGENT GGUF (ftype 114, protected output heads), with the repo's own MTP-Q4_0 head drafting four tokens at one native 262,144 window, f16 KV, the BF16 projector for vision, and the measured rejection of the ngram-mod chain."
-tags:
-  - llm
-  - frank-dragoon
-  - dankclaw
-  - llama-cpp
-  - rocmfpx
-  - gfx1151
-  - gguf
-  - spec-decode
-  - mtp
-timestamp: 2026-10-03
----
-
-# frank-dragoon — ROCmFPX Uncensored Q6_0_ROCMFPX_AGENT + MTP (gfx1151)
-
-> **RETIRED 2026-10-03 → 2026-10-05 (rollback target; payload, fork build and
-> build script all on disk).** Replaced by [[frank-dragoon-halogen27b-dflash2]]
-> (DFlash2, publisher 31.7 t/s mean vs the 27.6–28.1 measured here). This page
-> remains the contract record for the rollback. Original banner follows.
-> ACTIVE since 2026-10-03. Cut over from
-> [[frank-dragoon-rocmfpx-q8agent-mtp]] (ALIGNED Q8_0_ROCMFPX_AGENT, 17.0 t/s)
-> on Frank's request: the *uncensored* variant of the same protected-heads
-> family, one quant rung down. The swap turned out to be a strict upgrade on
-> the axis that matters here: **decode 27.6–28.1 t/s at a 4,813-token prompt
-> against 17.0 at a 2,202-token prompt** — faster with a deeper context, not a
-> same-depth comparison, but the sign is not in question.
-
-## Identity
-
-| | |
-|---|---|
-| Host / device | Dankclaw (192.168.0.143), Radeon 8060S iGPU `gfx1151`, ~114 GiB GTT aperture |
-| Endpoint | `http://192.168.0.143:7770` — alias `frank-dragoon`, `/v1` OpenAI surface |
-| Runtime | **`kingjones30/ROCmFPX`** llama.cpp fork @ `36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built by [`scripts/build-rocmfpx.sh`](../../scripts/build-rocmfpx.sh) — unchanged from the Q8 era; the fork already knew type 102 |
-| Unit | root systemd `frank-dragoon.service` ([`scripts/frank-dragoon.service`](../../scripts/frank-dragoon.service)), `User=dank`, `Restart=always` |
-| Serve script | [`scripts/serve-frank-dragoon.sh`](../../scripts/serve-frank-dragoon.sh) (owns the argv), tunables [`scripts/frank-dragoon.env`](../../scripts/frank-dragoon.env) |
-| Weights | `/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf` — 24,856,607,008 B, sha256 `e966deb3…` |
-| Draft | `…/mtp-Qwen3.8-27B-Q4_0.gguf` — 1,680,271,648 B, sha256 `051a1764…` (byte-identical to the aligned repo's head; hardlinked, not re-downloaded) |
-| Projector | `…/mmproj-Qwen3.8-27B-BF16.gguf` — 931,145,888 B (hardlinked from the aligned payload dir; the same file the Q8 era served) |
-| Fetch | [`scripts/download-frank-dragoon-kingjones-uncensored.sh`](../../scripts/download-frank-dragoon-kingjones-uncensored.sh) — hardlinks the draft/projector from the aligned dir when present, otherwise fetches; `-f`, byte-count + LFS sha256 verified, `--verify-only`, token never on the command line |
-| Seam | `tests/frank_dragoon_unit.bats` (payload gates, deployer/serve parity, spec block, KV dtype) |
-
-## Why this file
-
-`kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF` ships the same
-"protected heads" construction as the aligned repo, in the uncensored tuning.
-Parsed header of the deployed file:
-
-| Field | Value |
-|---|---|
-| ftype | **114** — the fork's 64-bit-block family, ≈7.44 bpw |
-| Tensor mix | 353 F32 + **339 type-102 (`GGML_TYPE_Q6_0_ROCMFPX`)** + **159 type-103 (`Q8_0_ROCMFPX`)** |
-| Protected heads | 64 output-head tensors kept at plain precision — the AGENT construction from the aligned repo, at the 6-bit rung |
-| Arch | qwen35, 64 layers, `full_attention_interval 4`, `context_length 262144` |
-| Embedded sampling recipe | temp 1.0 / top_p 0.95 / top_k 20 |
-
-The same lock-in law as the Q8 era applies: types 102 and 103 exist in no
-loader but this fork; stock llama.cpp and gufo refuse the file at load
-(`unknown tensor type`). The loader was not re-litigated — the file picks the
-runtime, and the pin in `scripts/build-rocmfpx.sh` (commit by SHA) stays.
-
-Rejected sibling: `Q6_0_ROCMFPX_LEAN` (6.45 bpw, no protected heads) — the
-protection is what keeps the output-head logits at full precision and the
-drafter accurate, so it stays deployed.
-
-## Shape and budget
-
-Unchanged from the Q8 era — the swap was payload-only:
-
-| Dial | Value | Why |
-|---|---|---|
-| `--ctx-size` | 262,144 | the native window, no rope scaling |
-| `--parallel` | 1 | one conversation, all memory to it |
-| KV dtype | **f16** (both K and V) | ~64 KiB/token; free inside a 114 GiB aperture |
-| `--fit off` | on | the auto-fitter would silently resize the window |
-| `--flash-attn on` | on | required for the long-window attention path |
-| `--direct-io`, `--ctx-checkpoints 16`, `--cache-ram 8192`, `--cache-idle-slots`, `--jinja`, `--reasoning on --reasoning-format deepseek --reasoning-budget -1` | | same as [[frank-dragoon-rocmfpx-q8agent-mtp]] |
-
-Spec decode: `--spec-type draft-mtp`, the repo's `mtp-Qwen3.8-27B-Q4_0.gguf`,
-`--spec-draft-n-max 4`, pinned `--spec-draft-device ROCm0`.
-
-Measured steady state: GTT **~61 GiB** under live load (host baseline 16.7),
-boot to `/health` **~15 s**.
-
-## Measured on this payload (2026-10-03, production service, `/completion` timings, temp 0, `cache_prompt: false`)
-
-| Traffic | Decode t/s | Prefill t/s | Draft acceptance |
-|---|---|---|---|
-| Prose, 4,813-token prompt, 512 tokens | **27.57 / 28.10** (two runs) | ~205–217 | **396/456 = 0.868** → mean acceptance length ≈ 4.4, i.e. drafting at nearly the full n-max 4 |
-| Schema echo (structured repetition) | **31.44** | — | 639/639 = 1.000 |
-| Block copy (text repeated verbatim) | ~28.0 | — | 0.848 |
-
-vs the Q8 era's 0.462 acceptance at a 2,202-token prompt: the MTP head drafts
-materially better on this file. The card's claimed 0.953-class acceptance is
-no longer science fiction on real prose — but keep reading it as a file
-property, not a throughput promise.
-
-## The ngram-mod chain — measured and rejected
-
-Frank's ask: layer `--spec-type draft-mtp,ngram-mod` on top of MTP for an
-extra gain. Measured in three bench windows (production service stopped per
-the benchmarking law; artifacts `/home/dank/fl-ngram-ab/` on Dankclaw),
-comparing `draft-mtp` alone vs the chain on the same traffic:
-
-| Traffic | MTP alone | MTP + ngram-mod | Delta |
-|---|---|---|---|
-| Prose | 27.57 t/s | 23.40 t/s | **−15 %** |
-| Block copy | 28.0 t/s | 26.1 t/s | **−7 %** (with ngram-mod demonstrably firing: 138 proposals/run) |
-
-Mechanism, straight from arm B's boot log:
-
-```
-recurrent state rollback is not compatible with 'ngram-mod' - disabling rollback support
-```
-
-On the qwen35 hybrid (Gated DeltaNet recurrent layers) adding the ngram layer
-disables the rollback path MTP's own verification relies on, so acceptance —
-and throughput — drop on *all* traffic, even where the ngram matcher fires.
-**Deployed decision: `draft-mtp` alone.** Chaining is not free on recurrent
-models; the fleet law in `AGENTS.md → llama.cpp serving` records it.
-
-## Bench-window hardening laws (learned building the A/B)
-
-1. **Health 200 is not proof of ownership.** After boot, verify the pid you
-   started owns the port (`ss -ltnp`); a zombie server from a prior window
-   answers `/health` too and silently serves your arm.
-2. **Kill scratch servers with SIGKILL after a grace period.** llama-server's
-   graceful SIGTERM waits on open keep-alive connections — a harness holding
-   the socket leaves the process (and its ~24 GiB of GTT) alive after the
-   script exits.
-3. **Prove the port is free and GTT dropped between arms**, or the second
-   arm measures a collision.
-
-## Client contract
-
-Unchanged from the Q8 era and re-verified on this payload 2026-10-03: model id
-`frank-dragoon`, native tools, thinking in `reasoning_content`,
-`reasoning_effort` ∈ {xhigh, medium, low} only (anything else HTTP 500),
-card-recipe sampling defaults server-side, vision through the BF16 projector
-(red PNG → **`Red`**, 74 prompt / 165 completion tokens). The traps list —
-`content: null` when reasoning eats the budget, the self-recovering MTP
-boundary log line, `enable_thinking:false` not `reasoning_effort:"none"` — is
-in [[frank-dragoon-rocmfpx-q8agent-mtp]] and still applies.
-
-Sweep key for scratch servers is now the payload name:
-`pkill -f 'Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT'` — never `pkill -f
-llama-server` (frank-zealot is also one).
-
-## Operations
-
-```bash
-./scripts/deploy.sh dankclaw --service frank-dragoon
-systemctl status frank-dragoon
-journalctl -u frank-dragoon -n 200 --no-pager
-scripts/download-frank-dragoon-kingjones-uncensored.sh --verify-only
-```
-
-## Rollback
-
-| Target | How |
-|---|---|
-| **ALIGNED Q8_0_ROCMFPX_AGENT** (previous shape, 17 t/s) | the payload dir `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/` is untouched on disk; revert the swap commit (unit/env/serve trio) and redeploy, or point `FRANK_DRAGOON_MODEL_FILE` + `LLM_DRAGOON_MODEL_FILE` at the old paths. Verify with `scripts/download-frank-dragoon-kingjones.sh --verify-only` |
-| gufo Q4_K_XL era | git tag `frank-dragoon-gufo-q4kxl-20261001`, **but the gufo payloads (UD-Q4_K_XL 16.35 GiB, UD-Q8_K_XL + DFlash2 drafter) were removed from disk 2026-10-03 with Frank's approval** — this rollback now requires re-download per [[frank-dragoon-gufo-qwen38-27b]] |
-
-## When it misbehaves
-
-Same decision tree as [[frank-dragoon-rocmfpx-q8agent-mtp]]:
-`unknown tensor type` ⇒ off the pinned fork; SIGSEGV in
-`ggml_compute_forward_set_rows` ⇒ pre-`36e9acd` build; slow decode with no
-error ⇒ draft not engaged (`draft_n` in `/completion` timings) or an orphaned
-second copy eating GTT (`/sys/class/drm/card1/device/mem_info_gtt_used`);
-500 naming Jinja ⇒ `--jinja` lost; 500 naming the template ⇒ unmapped
-`reasoning_effort` in `models.yml`.

serverAppConfigs-docs/LLM/frank-dragoon-rocmfpx-q8agent-mtp.md

@@ -1,232 +0,0 @@
----
-type: reference
-title: "frank-dragoon on Dankclaw — ROCmFPX llama.cpp, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (retired)"
-description: "The retired frank-dragoon runbook — the kingjones30/ROCmFPX llama.cpp fork (pinned commit, gfx1151) serving kingjones777's Q8_0_ROCMFPX_AGENT GGUF, a ftype-115 file no other loader can read, with the repo's own MTP-Q4_0 head, f16 KV, and the BF16 projector. Measured MTP-vs-none A/B, the client contract, and the rollback state as of 2026-10-03."
-tags:
-  - llm
-  - frank-dragoon
-  - dankclaw
-  - llama-cpp
-  - rocmfpx
-  - gfx1151
-  - gguf
-  - spec-decode
-  - mtp
-timestamp: 2026-10-02
----
-
-# frank-dragoon — ROCmFPX Q8_0_ROCMFPX_AGENT + MTP (gfx1151, retired)
-
-> **RETIRED 2026-10-03** — superseded by
-> [[frank-dragoon-rocmfpx-q6agent-uncensored-mtp]]: Frank swapped to the
-> uncensored Q6_0_ROCMFPX_AGENT file (protected heads, one rung down), which
-> measures **27.6–28.1 t/s** at a deeper prompt than this file's 17.0. This
-> runbook stays live as the rollback reference and the trap catalogue; the
-> payload it documents remains on disk at
-> `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/`.
->
-> Cut over 2026-10-02 from [[frank-dragoon-gufo-qwen38-27b]] (Q4_K_XL, ~15 t/s)
-> on two requests from Frank: *highest quality that runs*, and *one full
-> context*. It delivered 8.39 bpw weights — two quant rungs above what gufo
-> could load — at 17.0 t/s, so quality was not bought with speed.
->
-> The capability that forced this whole shape is a GGUF tensor type:
-> **`GGML_TYPE_Q8_0_ROCMFPX` (103)**. It exists in exactly one loader on this
-> host, so the runtime was decided by the file, not by preference.
-
-## Identity
-
-| | |
-|---|---|
-| Host / device | Dankclaw (192.168.0.143), Radeon 8060S iGPU `gfx1151`, ~114 GiB GTT aperture |
-| Endpoint | `http://192.168.0.143:7770` — alias `frank-dragoon`, `/v1` OpenAI surface |
-| Runtime | **`kingjones30/ROCmFPX`** llama.cpp fork @ `36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built by [`scripts/build-rocmfpx.sh`](../../scripts/build-rocmfpx.sh) → `/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server` |
-| Unit | root systemd `frank-dragoon.service` ([`scripts/frank-dragoon.service`](../../scripts/frank-dragoon.service)), `User=dank`, `Restart=always` |
-| Serve script | [`scripts/serve-frank-dragoon.sh`](../../scripts/serve-frank-dragoon.sh) (owns the argv), tunables [`scripts/frank-dragoon.env`](../../scripts/frank-dragoon.env) |
-| Weights | `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` — 28,214,425,952 B, sha256 `f3ef2bdbf56e244f…` |
-| Draft | `…/mtp-Qwen3.8-27B-Q4_0.gguf` — 1,680,271,648 B, sha256 `051a1764cff8c4f3…` |
-| Projector | `…/mmproj-Qwen3.8-27B-BF16.gguf` — 931,145,888 B, sha256 `de2a4986…` |
-| Fetch | [`scripts/download-frank-dragoon-kingjones.sh`](../../scripts/download-frank-dragoon-kingjones.sh) (`-f`, byte-count + LFS sha256 verified, `--verify-only`, token never on the command line) |
-| Seam | `tests/frank_dragoon_unit.bats` (19 tests: payload gates, deployer/serve parity, spec block, KV dtype, sweep identity) |
-
-## Why this file, and why only this runtime
-
-`kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` ships several rungs. Parsed
-headers of the two candidates:
-
-| File | ftype | Tensor mix | Output-head tensors at plain Q8_0 | Card MTP acceptance |
-|---|---|---|---|---|
-| `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` | **115** | 353 F32 + 187 Q8_0 + **311 type-103** | **187** | 0.953 |
-| `Qwen3.8-27B-Q8_0_ROCMFPX.gguf` | 111 | 2 Q8_0 + type-103 | 2 | 0.911 |
-
-Two consequences, both measured off the header rather than guessed:
-
-1. **Quality.** The AGENT routing variant keeps the output-head (and other
-   routing-sensitive) tensors at plain Q8_0 instead of folding them into the
-   ROCmFPX kernel layout — 187 vs 2 — so the logits the sampler reads are the
-   highest-precision tensors in the file. It is also the card's MTP-tuned
-   variant, so it is the *faster* drafter base at the same time.
-2. **The runtime is forced.** Type 103 is a private extension: stock llama.cpp
-   and gufo both fail at load (`unknown tensor type`). Verified on the host
-   before deploying — the bake-off was not a horse race, it was a lock-in
-   check.
-
-**The build identity matters as much as the fork.** The 2026-08-25 ROCmFPX
-builds read this file and then died in `ggml_compute_forward_set_rows` — the
-exact failure documented in [[kingjones-q8agent-rocmfpx]] when this payload was
-first attempted on Dankclaw and routed to Dankserver as
-[[frank-medium-rocmfpx]] instead. `scripts/build-rocmfpx.sh` therefore checks
-the commit out **by SHA** and refuses to build anything else; the pin is
-load-bearing, not tidiness.
-
-## Shape and budget
-
-One session, the model's full native window, unquantized KV:
-
-| Dial | Value | Why |
-|---|---|---|
-| `--ctx-size` | 262,144 | "one full context" — the native window, no rope scaling |
-| `--parallel` | 1 | one conversation, all memory to it |
-| KV dtype | **f16** (both K and V) | ~64 KiB/token → ~16 GiB; free inside a 114 GiB aperture, so there is no reason to pay a quant |
-| `--fit off` | on | the auto-fitter would silently resize the window we asked for |
-| `--flash-attn on` | on | required for the long-window attention path to fit |
-| `--direct-io` | on | bypasses page cache for the 26 GiB weight stream (mmap is disabled by it — deliberate) |
-| `--ctx-checkpoints 16` | 16 | slot state for checkpoint/rollback |
-| `--cache-ram 8192` + `--cache-idle-slots` | 8 GiB | prompt cache; verified a repeat pass over the same 6k prompt replays in **0.5 s** |
-| `--jinja` | on | without it `chat_template_kwargs` are silently ignored |
-| `--reasoning on --reasoning-format deepseek --reasoning-budget -1` | | thinking in its own channel, unbounded |
-
-Measured steady state (production, one model copy resident):
-`mem_info_gtt_used` **63.1 GiB** of the ~114 GiB aperture, `mem_info_vram_used`
-0.17 / 2.0 GiB (the iGPU's dedicated carve-out is a stub; everything real lives
-in GTT). llama-server RSS 2.1 GiB. Boot to `/health` 200: **16 s** — the fastest
-endpoint bring-up in the fleet, because there is no engine compile step.
-
-## Speculative decode — measured, not assumed
-
-```
---spec-type draft-mtp --model-draft <mtp-Qwen3.8-27B-Q4_0.gguf>
---spec-draft-n-max 4 --spec-draft-n-min 0 --spec-draft-p-min 0.0
---spec-draft-device ROCm0 --spec-draft-ngl 99
-```
-
-MTP in llama.cpp is a property of the file, not of the flags: the head is a
-trained tensor set (`nextn_predict_layers = 1` in this repo's head, arch
-`qwen35`, 65 blocks) and the draft context runs on the main model's weights, so
-there is no second 26 GiB copy in memory. See the fleet law in
-`AGENTS.md → llama.cpp serving`.
-
-**A/B, 2026-10-02** — same session, temp 0, 2,202-token prompt, 512 tokens,
-`cache_prompt: false`, `frank-dragoon.service` stopped for every scratch arm (the
-benchmarking law), baseline arm stripped of `--model-draft` **and** every
-`--spec-*` flag:
-
-| Arm | Decode t/s (2 runs) | Prefill t/s | Draft acceptance |
-|---|---|---|---|
-| **MTP, n-max 4 (deployed)** | **17.00 / 17.14** | 350.6 / 324.9 | 0.462 (mean acceptance length ≈ 1.86) |
-| no draft | 7.60 / 7.66 | 381.8 / 381.0 | — |
-
-**+123 % decode for an 8–15 % prefill tax.** Notes that matter before someone
-re-litigates this:
-
-- Acceptance here (0.462) is nowhere near the card's 0.953. The card measures
-  its own short, repetitive text; this is a 2.2k-token prose prompt. Treat the
-  card's number as a file property, not a throughput promise.
-- `n-max 4` is where the gain was measured; 2 and 8 are **unmeasured**. The
-  gufo-era law applies in spirit — never rank spec settings on a
-  self-predictable corpus.
-- End-to-end on the chat path is slower than the `/completion` number because
-  thinking tokens are included: a 2,253-token chat prompt generating 512
-  tokens took 43.8 s wall (11.7 t/s end-to-end) — that is the cost of an
-  unbounded reasoning budget, not a regression in decoding.
-
-## Client contract (live-verified 2026-10-02)
-
-| Behavior | Observation |
-|---|---|
-| Model id | `/v1/models` → `frank-dragoon` |
-| Tools | native: `get_weather` → `{"city": "Seattle"}` in `tool_calls` |
-| Thinking | in **`reasoning_content`** (llama.cpp field — *not* vLLM's `reasoning`); `models.yml` sets `reasoningContentField` accordingly |
-| `reasoning_effort` | accepts **only** `xhigh`, `medium`, `low`; anything else is HTTP 500 (the template raises, it does not clamp). `models.yml` folds `high`/`max` → `xhigh` |
-| Sampling | server defaults = the card's thinking recipe (temp 1.0, top_p 0.95, top_k 20, min_p 0, presence 0), mirrored in the harness `extraBody` |
-| Vision | the BF16 projector loads (`loaded multimodal model`); a red square answers **`Red`** (`usage: 75 prompt / 71 cached`) |
-| Prompt cache | second pass over the same 6k prompt: 0.5 s wall |
-
-Three traps, all hit for real:
-
-1. **`content: null` when reasoning eats the budget.** With
-   `--reasoning-parser`-style separation, a request whose `max_tokens` is
-   consumed entirely by thinking returns `message.content = None`. A bench
-   client that indexes `content` crashes *after* measuring and loses the
-   numbers — fall back to `reasoning_content`, or size the budget (200 was
-   enough for the vision probe).
-2. **`MTP boundary missing for seq_id=… ; resyncing after a non-token step`**
-   appears in the log under draft resets. Self-recovering; not an outage signal.
-3. **Never `pkill -f llama-server` on Dankclaw** — frank-zealot is also a
-   `llama-server`. The sweep key here is the payload name:
-   `pkill -f 'Qwen3.8-27B-Q8_0_ROCMFPX_AGENT'`.
-
-## Operations
-
-```bash
-# deploy (planner selects this endpoint only via its own unit/env/serve trio)
-./scripts/deploy.sh dankclaw --service frank-dragoon
-
-# status / logs / cause of a crash loop
-systemctl status frank-dragoon
-journalctl -u frank-dragoon -n 200 --no-pager
-
-# rebuild the pinned fork (only when deliberately moving the pin)
-scripts/build-rocmfpx.sh
-
-# re-verify the payload without re-downloading
-scripts/download-frank-dragoon-kingjones.sh --verify-only
-```
-
-`serve-frank-dragoon.sh` refuses to start — before touching the port — unless the
-fork binary, the weights, the MTP head and (unless `FRANK_DRAGOON_MMPROJ=""`) the
-projector all exist; `scripts/deploy-remote.sh` mirrors the same predicate so a
-broken payload never stops the running unit. Caller environment **overrides**
-`frank-dragoon.env`, and defaults use `:=` so an explicitly empty value surfaces
-as the error it reads as.
-
-## Rollback (as of retirement, 2026-10-03)
-
-**Primary rollback: the ALIGNED Q8 file itself is the previous shape's
-payload and remains on disk** at
-`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/` — revert the
-2026-10-03 swap commit (unit/env/serve trio) or repoint
-`FRANK_DRAGOON_MODEL_FILE`/`LLM_DRAGOON_MODEL_FILE`, verify with
-`scripts/download-frank-dragoon-kingjones.sh --verify-only`, redeploy.
-
-git tag **`frank-dragoon-gufo-q4kxl-20261001`** (gufo v0.1.1, Q4_K_XL +
-DFlash2) is now a **re-download** rollback: the gufo payloads
-`/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL/` (16.35 GiB) and
-`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/` (incl. the DFlash2 drafter) were
-removed from disk on 2026-10-03 with Frank's explicit approval to free space.
-`scripts/build-gufo.sh` remains on disk; re-fetch per
-[[frank-dragoon-gufo-qwen38-27b]].
-
-The one-day vLLM MiMo era's payloads (MiMo-V2.6 BF16 18 GiB,
-Qwen3.5-9B MTP source 19 GiB, both DFlash draft dirs) were staged to
-`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 —
-recoverable until the trash is purged, and never part of a rollback plan.
-
-## When it misbehaves
-
-- **`unknown tensor type` / ftype 115 refused** — something moved off the
-  ROCmFPX binary. Check `readlink -f $(systemctl show frank-dragoon -p ExecStart)`
-  and the fork commit; this file cannot run on stock llama.cpp or gufo.
-- **SIGSEGV in `ggml_compute_forward_set_rows` during load** — a fork build
-  older than the `36e9acd` pin; rebuild with `scripts/build-rocmfpx.sh`.
-  History: [[kingjones-q8agent-rocmfpx]].
-- **Slow decode with no log error** — check whether the draft is engaged
-  (`/metrics`, `draft_n` / `draft_n_accepted`; a zero draft count means the head
-  did not load), and check GTT pressure with
-  `cat /sys/class/drm/card1/device/mem_info_gtt_used` — a second resident copy
-  of this model (an orphaned scratch server) drives the whole host into
-  eviction and halves the rate silently.
-- **`500` naming Jinja** — a `chat_template_kwargs` field reached a server
-  without `--jinja`.
-- **HTTP 500 naming the template on the first request of a role** — an
-  unmapped `reasoning_effort`; fix the fold in `models.yml`, not the model.

serverAppConfigs-docs/LLM/frank-dragoon-unsloth-qwen38-27b-q8xl.md

@@ -1,117 +0,0 @@
----
-type: reference
-title: "stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-dragoon)"
-description: "The RETIRED frank-dragoon runbook (2026-09-21 → 2026-09-28; replaced by [[frank-dragoon-gufo-qwen38-27b]], and the rollback target) — the stock llama.cpp prebuilt (unsloth b11030, ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.46 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) and q8_0 KV on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the full-window/one-stream shape, the DFlash2 spec method, the Qwen-Sharp template, the payload, the client contract, and the failure modes worth knowing."
-tags:
-  - llm
-  - llama.cpp
-  - unsloth
-  - dankclaw
-  - frank-dragoon
-  - strix-halo
-  - gfx1151
-  - q8
-  - dflash2
-  - rocm
-  - q8-kv
-timestamp: 2026-09-21
----
-
-# frank-dragoon — stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2)
-
-> **RETIRED 2026-09-28**, replaced by [[frank-dragoon-gufo-qwen38-27b]] (the
-> gufo engine, same weights, same port and alias); the prebuilt is intact on
-> disk as the rollback target. ACTIVE from 2026-09-21, when it replaced
-> [[frank-dragoon-kairic-edge-qwen38-27b]] (Kairic Edge IU4 W4A4) and
-> [[frank-dragoon-unsloth-qwen38-flash-next]] (the Flash-Next MoE it
-> preceded).
-
-## Identity
-
-- **Endpoint:** `frank-dragoon` at `192.168.0.143:7770` (Dankclaw).
-- **Runtime:** stock llama.cpp — the unsloth `b11030` prebuilt
-  (`/home/dank/GitPackages/unsloth-llama-b11030/llama-server`), ROCm gfx1151.
-  A mainline build, not a fork: it supports the `qwen3_5` architecture and the
-  DFlash2 drafter (`llama_model_qwen35`, `llama_model_dflash`,
-  `common_speculative_impl_draft_dflash`).
-- **Service shape:** root systemd unit `scripts/frank-dragoon.service`
-  (`User=dank`), launched by `scripts/serve-frank-dragoon.sh` with tunables from
-  `scripts/frank-dragoon.env`.
-
-## Shape
-
-- **Model:** `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` — a 27B **dense** model,
-  31.46 GiB GGUF (sha256-verified by `scripts/download-frank-dragoon.sh`).
-- **Context:** one stream, full 262,144 tokens (the model's native window),
-  `PARALLEL_SLOTS=1`.
-- **Placement:** all layers on the iGPU (`--n-gpu-layers 999`). The 27B dense
-  weights (31.46 GiB) + the DFlash2 drafter (1.14 GiB) + q8_0 KV (~8.6 GiB) +
-  compute buffers fit the iGPU's GTT aperture (~114 GiB).
-
-## Spec method
-
-DFlash2 (a block-diffusion drafter) at **depth 4**. The drafter is the z-lab
-`Qwen3.8-27B-DFlash2-Q4_K_M` (1.14 GiB), loaded via `--model-draft` with
-`--spec-type draft-dflash` and `--spec-draft-device ROCm0`. It has a sliding
-window of 2048, so its draft KV is small (~40 MiB). Greedy verification
-(`--temp 0 --top-p 1.0 --top-k 0 --min-p 0.0`) is required.
-
-## KV
-
-**q8_0** (`--cache-type-k q8_0 --cache-type-v q8_0`), ~8.6 GiB at 262k. The
-unsloth b11030 build does **not** support fp8 KV (allowed: f32/f16/bf16/q8_0/
-q4_0/q4_1/iq4_nl/q5_0/q5_1); q8_0 is the highest-precision 8-bit option.
-
-## Template
-
-The Qwen-Sharp v22.4.0 template is pinned via `--chat-template-file`
-(`dankserver/frank-medium/qwen-sharp-v22.4.0.jinja`) with `--jinja` and
-`--chat-template-kwargs '{"preserve_thinking":true}'`. Without
-`--chat-template-file` the binary falls back to the built-in `peg-native`
-template, which produces a ~73,600-token prompt for a ~250-token request.
-`preserve_thinking` keeps prior thinking blocks across turns for multi-turn
-agents. The template gates `reasoning_effort` to `{xhigh, medium, low}` — the
-harness folds its levels onto that set in `models.yml` (`reasoningEffortMap`).
-
-## Payload
-
-`scripts/download-frank-dragoon.sh` fetches the main GGUF and the DFlash2
-drafter, each verified against its declared byte count + sha256
-(`--verify-only` re-checks). The serve script refuses to start if the binary,
-the main GGUF, or the drafter is missing.
-
-## Client contract
-
-- **Alias:** `frank-dragoon` (the `id` at `/v1/models`).
-- **Reasoning:** extracted to `reasoning_content` (`--reasoning auto`,
-  `--reasoning-format deepseek`, unlimited budget).
-- **Tools:** supported (`--jinja` is pinned).
-- **Vision:** none — text-only (no mmproj), matching the prior frank-dragoon.
-- **Sampling:** greedy by default; a client that sends a temperature gets its
-  own, and the harness applies per-model overrides from `models.yml`.
-
-## Failure modes worth knowing
-
-- **Model load is slow (minutes cold).** Health returns 503 while loading; do
-  not kill the process. `deploy.sh` waits up to 20 minutes (`LLM_WAIT`).
-- **`--ngl` (double-dash) is invalid in b11030** — use `--n-gpu-layers`
-  (the serve script uses the long form).
-- **The 8-bit UD-Q8_K_XL decodes ~17 t/s** vs the prior 4-bit Kairic Edge IU4
-  at ~22 t/s — the ~23 % larger weight size (31.46 vs 26 GiB) explains the
-  gap; it is not a regression.
-- **DFlash2 draft acceptance is text-variety-dependent** (0.85 on a trivial
-  one-word answer, 0.519 on a varied explanatory prompt) — do not read a
-  single-request acceptance as the steady state.
-
-## Rollback
-
-Payload and runtime for [[frank-dragoon-kairic-edge-qwen38-27b]] (Kairic Edge
-IU4 W4A4) remain intact on disk. See [[frank-dragoon-unsloth-qwen38-flash-next]]
-for the Flash-Next predecessor it replaced on 2026-09-21.
-
-# Citations
-
-- unsloth prebuilts: https://huggingface.co/unsloth
-- Model: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
-- DFlash2 drafter: https://huggingface.co/z-lab
-- Serve script: `scripts/serve-frank-dragoon.sh`; tunables: `scripts/frank-dragoon.env`

serverAppConfigs-docs/LLM/frank-dragoon-unsloth-qwen38-flash-next.md

@@ -1,47 +0,0 @@
----
-type: reference
-title: "unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on Dankclaw (frank-dragoon)"
-description: "The RETIRED frank-dragoon runbook (2026-09-21, same-day swap; replaced by [[frank-dragoon-unsloth-qwen38-27b-q8xl]]) — the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the 200k/one-stream shape, the explicit -md MTP wiring, the payload, the client contract, and the failure modes worth knowing."
-tags:
-  - llm
-  - unsloth
-  - dankclaw
-  - frank-dragoon
-  - strix-halo
-  - gfx1151
-  - qwen4exp
-  - rocm
-  - mtp
-timestamp: 2026-09-21
----
-
-# frank-dragoon — unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS (RETIRED)
-
-> **RETIRED 2026-09-21 (same-day swap).** This was the ACTIVE frank-dragoon
-> runbook on 2026-09-21 before it was replaced by
-> [[frank-dragoon-unsloth-qwen38-27b-q8xl]] (27B dense UD-Q8_K_XL) later the
-> same day. The payload and runtime remain on disk as the rollback target.
-
-## What it served
-
-- **Runtime:** unsloth llama.cpp prebuilt `b11030-mix-5ff778e` (ROCm
-  gfx1151), root systemd unit, port 7770.
-- **Model:** `unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS` (a 125B MoE / 6B
-  active), 87.3 GiB shipped as three split parts. The split parts ARE the
-  payload — concatenating them trips `invalid split file name` (part 1's
-  header advertises `file_count=3`).
-- **Shape:** one 200,000-token conversation (single stream), MTP depth 2 via
-  the explicit `-md` shared-Q8_0 head. Draft acceptance verified live
-  0.83–0.92.
-- **Memory:** the 114 GiB GTT aperture on the Strix Halo iGPU.
-
-## Rollback
-
-Payload and runtime intact on disk. See [[frank-dragoon-unsloth-qwen38-27b-q8xl]]
-for the current ACTIVE runbook and [[frank-dragoon-ciru-orca]] for the
-predecessor.
-
-# Citations
-
-- unsloth prebuilts: https://huggingface.co/unsloth
-- Model: https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF

serverAppConfigs-docs/LLM/frank-dragoon-vllm-mimo-v26-bf16.md

@@ -1,186 +0,0 @@
----
-type: reference
-title: "frank-dragoon on Dankclaw — vLLM TheRock, MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (RETIRED)"
-description: "The RETIRED frank-dragoon runbook (2026-10-01, one day) — the digest-pinned kyuz0 TheRock gfx1151 vLLM image serving XiaomiMiMo MiMo-V2.6-Distill-Qwen-9B in full BF16 on the Strix Halo iGPU. Kept for the measured record: the roof math, the cross-model MTP drafter, and the autoresearch finding that spec decode is NET-NEGATIVE on real chat+reasoning traffic here (6.19 t/s with MTP-2 vs 9.01 t/s without). Rolled back to [[frank-dragoon-gufo-qwen38-27b]] by Frank the same night."
-tags:
-  - llm
-  - frank-dragoon
-  - dankclaw
-  - vllm
-  - strix-halo
-  - mimo
-  - mtp
-  - spec-decode
-  - bf16
-timestamp: 2026-10-01
----
-
-# frank-dragoon — vLLM MiMo-V2.6-Distill-Qwen-9B BF16 (cross-model MTP)
-
-> **RETIRED 2026-10-01 night, one day active.** Frank's call after the
-> autoresearch loop: battery decode 6.19 t/s (MTP-2) / 9.01 t/s (no draft)
-> against gufo's ~15 t/s on the 27B — and every ≥6-bit quant path in vLLM
-> 0.30 is dead on gfx1151 (quark int8 rejected; all FP8 kernels CDNA-only;
-> torchao config unwired). Back to [[frank-dragoon-gufo-qwen38-27b]]
-> (Q4_K_XL, 1 x 262144). The measured content below stands as the record.
-
-## Identity
-
-- **Endpoint:** `frank-dragoon` at `192.168.0.143:7770` (Dankclaw).
-- **Runtime:** vLLM 0.30.0 in the **digest-pinned** `docker.io/kyuz0/vllm-therock-gfx1151`
-  image (image ID `2d24a8c5a3fd`, digest `sha256:98c1f3ac9833…` — ROCm 10.0.0 /
-  torch 2.11.0 / TheRock, Triton Gated DeltaNet kernels). The official vLLM
-  images are CUDA-only; this fork is the fleet's gfx1151 path. Plain
-  `podman run` driven by `scripts/serve-frank-dragoon.sh` under a root unit
-  (`scripts/frank-dragoon.service`, `User=dank`, `Restart=always`,
-  `ExecStop=podman stop -t 120`, `TimeoutStopSec=180` > the stop grace).
-- **Model:** `XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` in **full BF16**
-  (17.53 GiB, 4 shards) — the same checkpoint frank-zealot serves as an INT8
-  W8A16 AutoRound conversion on the 3090, here unquantized.
-- **Draft:** `Qwen/Qwen3.5-9B` (18 GiB on disk; only the 15 `mtp.*` tensors
-  are loaded into the draft pass).
-
-## Why this shape (the roof math)
-
-Decode speed for a dense model on Strix Halo is bandwidth-bound:
-BF16 weights x one pass per token against the LPDDR5x aperture. Measured
-2026-10-01 on this host: **13.1 t/s is ~94% of the roof** — the payload, not
-a misconfiguration. The same INT8 payload reaches 83.9 t/s on the 3090
-because it moves ~12 GiB through ~936 GB/s. The only way past a bandwidth
-wall is generating more than one token per weight pass: **speculative decode
-is the performance feature of this endpoint**, not a garnish.
-
-The memory side is the opposite of a squeeze: KV grows ONLY on the 8 full
-attention layers (32 KiB/token bf16: 8 x 2 x 4 heads x 256 dim x 2 B);
-the 24 Gated DeltaNet layers hold a FIXED ~25 MiB state per running
-sequence. `--max-model-len 262144` (the native window) with
-`--max-num-seqs 4` at `--gpu-memory-utilization 0.45` boots clean — the
-four-session contract the gufo shape ran, with the pool shared by the
-scheduler instead of carved per session.
-
-## Spec method: cross-model MTP — measured, not assumed
-
-| arm | prefill | decode | acceptance |
-|---|---|---|---|
-| BF16 base | 17,876 t/s | 13.1 t/s | — |
-| BF16 + DFlash 8 tok | 7,976 t/s | 17.4 t/s | pos0 59% pos1 26% (token rate 14.8%) |
-| **BF16 + MTP 2 tok** | 7,380 t/s | **19.8 t/s** | pos0 71% pos1 44% (token rate 57.4%) |
-
-One session, temperature 0, 28,647-token prefill / 512-token decode, same
-client for all three arms (2026-10-01 bench window, frank-dragoon stopped per
-Benchmarking Policy). **MTP wins (+51% vs +33%) and needs only 2 draft
-tokens** — acceptance past position 1 is 44% and falling; DFlash's own curve
-decays 59/26/14/7, so 8 proposals are mostly wasted verification.
-`num_speculative_tokens` beyond 2 is UNMEASURED; do not raise it without a
-fresh A/B. Both spec arms pay ~55–59% of prefill (the draft passes share
-the compute queues) — this endpoint's thinking-heavy traffic spends its wall
-clock in decode, which is the regime where that trade wins.
-
-**Why a Qwen head accepts on a MiMo target:** MiMo-V2.6 is a distillation of
-Qwen3.5-9B that preserves the hidden-state geometry — identical
-4096-hidden/248320-vocab/32-layer/interval-4 structure, and the 71%/44%
-per-position acceptance is the direct measurement that the distillation
-preserved more than geometry. MiMo itself ships **ZERO `mtp.*` tensors**
-(verified in both the BF16 base and the INT8 conversion) — which is exactly
-why frank-zealot's SGLang NEXTN draft is a 0.00-acceptance ghost: SGLang
-reads the head from the TARGET checkpoint, vLLM reads the draft from a
-separate path. That difference is the whole trick:
-
-```
---speculative-config '{"method":"mtp","model":"/models/Qwen3.5-9B","num_speculative_tokens":2}'
-```
-
-The alternative drafter (`z-lab/Qwen3.5-9B-DFlash`, 6 layers, target_layer_ids
-[1,5,…,29]) stays fetched on disk; flip `FRANK_DRAGOON_SPEC=dflash` and repoint
-`FRANK_DRAGOON_SPEC_DIR` to use it. Its card's "vLLM PR #40898 unmerged" claim
-is FALSE — `vllm/v1/spec_decode/dflash.py` ships in 0.30.
-
-## The numerics trap (greedy text is NOT preserved by spec decode)
-
-Measured: temperature-0 completions DIFFER across arms (sha `1318aeb6` base,
-`0208b45d` dflash, `20c41c38` mtp). Batched verification changes the bf16
-reduction order, argmax near-ties diverge, and the divergence cascades.
-vLLM spec decode is distribution-preserving, not bitwise-greedy-preserving.
-The gufo-era "greedy == autoregressive text" law does not carry over; do not
-chase text drift between spec and no-spec arms as a regression.
-
-## Payload
-
-- Target: 4 safetensors shards (17.53 GiB) + config/tokenizer under
-  `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-BF16/`, each shard
-  byte-count + LFS sha256-verified by
-  `old/download-frank-dragoon-mimo.sh` (`--verify-only` re-checks; no token on
-  the curl command line).
-- MTP source: `old/download-frank-dragoon-qwen35.sh` →
-  `/home/dank/models/Qwen3.5-9B/` (4 shards behind
-  `model.safetensors.index.json`; the shards carry the odd HF asset names
-  `model.safetensors-0000N-of-00004.safetensors` — the index resolves them,
-  the engine boots from exactly this tree).
-- DFlash drafter: `old/download-frank-dragoon-dflash.sh` →
-  `/home/dank/models/Qwen3.5-9B-DFlash/`.
-- **Payload status 2026-10-02:** all three trees moved to
-  `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` (59 GiB staged with
-  the other retired frank payloads) and the fetch trio moved to `old/` — the
-  endpoint is two runtimes past this one, and re-fetch is cheaper than disk.
-- The serve script gates the image by REPO DIGEST (rootless podman resolves
-  tags from the network; the pin exists to boot exactly these bytes
-  offline), then config + index + exact shard count, then the spec draft
-  (either weight layout: single `model.safetensors` or a sharded index),
-  then `/dev/kfd` + whole `/dev/dri` (never individual renderD nodes —
-  Dankserver 2026-09-29 lesson).
-
-## Client contract
-
-- **Alias:** `frank-dragoon` (the `id` at `/v1/models`).
-- **Reasoning:** `--reasoning-parser qwen3`; vLLM emits the thinking text in
-  **`reasoning`**, NOT llama.cpp/gufo's `reasoning_content` — the harness
-  `models.yml` compat block for this provider sets
-  `reasoningContentField: "reasoning"`. Miss that and thinking silently
-  vanishes from transcripts.
-- **Tools:** `--enable-auto-tool-choice --tool-call-parser qwen3_coder`.
-- **Vision:** the checkpoint carries the `qwen3_5_vision` tower but the
-  serve script passes `--language-model-only` — the endpoint stays
-  text-only by decision; dropping the flag brings vision back.
-- **Sampling:** request-side; the harness sends the Qwen3.5-family recipe
-  (temp 0.7, top_p 0.8) via `compat.extraBody`, and
-  `chat_template_kwargs.{enable_thinking,preserve_thinking}` — the
-  `chat_template_kwargs` form is what vLLM reads; a top-level
-  `preserve_thinking` is ignored there.
-- **Env hygiene:** every tunable is `FRANK_DRAGOON_`-prefixed because HIP
-  reads bare `TEMP`/`HOST`/`GPU_MAX_HEAP_SIZE` and the serve script exports
-  the env file with `set -a`.
-
-## Ops
-
-- Boot ~200–240 s warm; health 503 while loading (do not kill it).
-- The serve script sweeps stale container identities before starting:
-  `frank-dragoon-mimo` itself and the bench-window container `vllm-mimo`.
-- Container death → `podman wait` returns non-zero from ExecStart →
-  `Restart=always` self-heals (a clean exit-0 drain also restarts, the
-  frank-archon lesson).
-- Bench windows: `systemctl stop frank-dragoon` is enough (mask fails on the
-  relinked unit; the deploy restores).
-- First-deploy facts (2026-10-01, deploy #3): healthy after **230 s**;
-  boot log `GPU KV cache size: 819,332 tokens, Maximum concurrency for
-  262,144 tokens per request: 3.13x` — four sessions share the pool
-  (the scheduler, not a carve). Live battery green: alias `frank-dragoon`,
-  reasoning split into `reasoning`, tool_calls with correct args, and
-  `/metrics` acceptance pos0 69% pos1 44% (54/86 draft tokens accepted),
-  matching the bench window.
-
-## Rollback
-
-Git tag **`frank-dragoon-gufo-q4kxl-20261001`** — gufo v0.1.1 (built from
-source, `scripts/build-gufo.sh` stays on disk) + the Q4_K_XL Qwen3.8-27B
-payload + DFlash2 drafter, all still on the host. Restoring the tag's
-unit/env/serve trio and redeploying returns the 27B endpoint.
-
-# Citations
-
-- Model: https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B
-- Draft source: https://huggingface.co/Qwen/Qwen3.5-9B
-- DFlash drafter: https://huggingface.co/z-lab/Qwen3.5-9B-DFlash
-- Engine image: https://hub.docker.com/r/kyuz0/vllm-therock-gfx1151
-- Serve script: `scripts/serve-frank-dragoon.sh`; tunables: `scripts/frank-dragoon.env`
-- Predecessor: [[frank-dragoon-gufo-qwen38-27b]]; sibling on the same arch:
-  [[frank-zealot-sglang-mimo-v26]]

serverAppConfigs-docs/LLM/frank-dragoon.md

@@ -0,0 +1,236 @@
+---
+type: reference
+title: "frank-dragoon on Dankclaw — halogen-flash-server 0.16.2, native v2 abliterated Qwen3.8-Flash-Next (canonical)"
+description: "The one runbook for frank-dragoon, the Strix Halo iGPU endpoint (192.168.0.143:7770): current shape from the env, every retired era with its rollback path, the durable laws this endpoint paid for, the wire contract, and the files that own it."
+tags:
+  - llm
+  - frank-dragoon
+  - dankclaw
+  - strix-halo
+  - gfx1151
+  - halogen
+timestamp: 2026-10-18
+---
+
+# frank-dragoon — halogen-flash-server 0.16.2, native v2 abliterated (Dankclaw iGPU)
+
+Strix Halo (gfx1151) endpoint `http://192.168.0.143:7770`, served id `frank-dragoon`: a podman container
+under a root `User=dank` unit. Live shape since 2026-10-17, pool raised to 401,408 on 2026-10-18
+(`2b12716`). This page replaces every per-era dragoon runbook; the Eras table is the rollback record.
+
+## Current shape
+
+`scripts/frank-dragoon.env` is authoritative and `scripts/serve-frank-dragoon.sh` passes exactly those
+flags and nothing else. Payload `/home/dank/models/halogen-v2-abliterated/`, verified by
+`scripts/download-frank-dragoon-v2abl.sh --verify-only`:
+
+| File | Bytes | sha256 | Source |
+|---|---|---|---|
+| `qwen38-flash-next-v2-abliterated.hgn` | 66,687,678,432 | `b5bd925f…` | Quat3rnion `@main` — the NATIVE 2026-10-11 v2 conversion of orcarouter's BF16 Qwen3.8-Flash-Next, abliterated |
+| `qwen38-flash-next-ngram.hgn` | 51,200,246,144 | `3450acada…` | peonist `@5cc17cea`, **REQUIRED**, hardlinked |
+| `qwen38-flash-next-vision.hgn` | 897,916,416 | `d62e0ae5…` | peonist `@e053f488`, hardlinked — vision ON |
+| `tokenizer/` (6 flat files) | ~22 MB | `tokenizer.json` LFS | peonist `@e053f488`, hardlinked |
+
+Image `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a`
+(0.16.2, closed source), container `frank-dragoon-halogen`, published `7770:8731`; the engine's
+unauthenticated token port 8730 stays on container loopback. Engine stdout →
+`/home/dank/logs/frank_dragoon.log`, **not** the journal.
+
+- Shape: `HALOGEN_CTX=200704` hard per-request ceiling · `HALOGEN_KV_SLOTS=2` ·
+  `HALOGEN_KV_POOL_POSITIONS=401408` = slots × ctx, the top of the band `tests/frank_dragoon_unit.bats`
+  enforces (`ctx ≤ pool ≤ slots × ctx`) · KV ~28 KiB/position → **11.0 GiB reserved** · no KV-dtype knob
+  exists here, the pool is the only KV lever · native window 262,144, `HALOGEN_ROPE_YARN` beyond it.
+- Knobs: `HALOGEN_MAX_TOK=8192` sizes the prefill arena (5.7 GiB) and is only the default for requests
+  omitting `max_tokens` · `HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` are a coupled
+  pair · MTP depth 2 with the head INSIDE the checkpoint (**`HALOGEN_MTP_HEAD` stays unset**) and
+  `HALOGEN_SPEC_ADAPT=0` · sampler 1.0 / 0.95 / 20 server-side · `HALOGEN_WEIGHTS_LOCK=1` ·
+  `HALOGEN_HOST_RESERVE_GIB=20` · prompt cache ON by image default (`PROMPT_CACHE=2`, no flag
+  passed): 12 entries, which the boot ledger prices at 111 MiB each at this window = 1.3 GiB
+  (the env header's "~85 MiB here" estimate did not survive that ledger line).
+- Deployed boot ledger: `62.9 GiB of weights locked in RAM (0.8 of them the vision tower's), 11.0 GiB
+  of KV pool, 5.7 GiB of working memory, 79.6 GiB in all`; n-gram `mapped 51.2 GB, NOT pinned
+  (Pin::None)`; no clamp line. Host `MemAvailable` read 22 GiB right after the 2026-10-18 raise —
+  ~20 GiB above the 20 GiB reserve, against 10–15 GiB of baseline co-tenancy (frank-zealot, ComfyUI,
+  Caddy/observability).
+- Harness (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw)`): `contextWindow: 155648`
+  (= ceiling − 45k per the 400-zone law, never higher), `maxTokens: 45056`, `input: [text, image]`,
+  `reasoningContentField: "reasoning_content"`, `high→xhigh` fold, `extraBody` 1.0/0.95. Compaction
+  rides [[frank-zealot]].
+- Deploy `./scripts/deploy.sh dankclaw --service frank-dragoon`; gate = boot-ledger greps (weights,
+  pool, MTP, prompt cache, **no clamp**) + `/health` 200 + a tool call + the 512 px red-PNG probe.
+  Kernel: `CONFIG_HSA_AMD_SVM=y` (CachyOS ships it) and `amdgpu.noretry=0` — the env says check it at
+  deploy; boot line per [[dankclaw-iommu-translation-off]].
+
+## Eras
+
+Rollback = revert that era's swap commit (or repoint the payload vars) and redeploy
+`dankclaw --service frank-dragoon`. Newest first; payloads staged under `~/.AGENTTRASH/` are **not** a
+rollback plan — recoverable only until the trash is purged.
+
+| Window | Payload | Runtime | Shape | Replaced because | Rollback |
+|---|---|---|---|---|---|
+| 2026-10-06 → 10-17 | Swift1.5 abliterated v2 `.hgn` r2 66,687,678,464 B `9c880b8b…` + same trio | flash-server 0.16.2, same digest | 2 slots, ctx 262,144, pool 524,288 (14.6 GiB), MTP always-on | 2026-10-17 A/B at that shape: NIAH 18/18 vs 17/18, refusal census 0/5 on both, tool calls + red-PNG probe identical, needle hit at both depths of a 21.5k haystack, decode 50.0 vs 46.8 t/s, cold prefill 1,604 vs 1,425 t/s, PPL 3.968 vs 3.877 (finetune's edge gone: +2.3 %), thinking 1,476 vs 784 mean, 7.4 GiB RAM back | `/home/dank/models/halogen-flashnext-v2/` + `scripts/download-frank-dragoon-flashnext.sh` |
+| 2026-10-05 → 10-06 | peonist `halogen-qwen3.8-27b` `qwen3.8-27b-p1w4d-d2.hgn` 35,865,565,184 B `274c3fc7…` | halogen 0.1.4 `sha256:dc0a39a0…` | 1 slot × 200,700, DFlash2 drafter, `HALOGEN_CACHE_MB=16384`, text-only, censored base | flash-server v2 restored vision, abliteration and MTP | `/home/dank/models/halogen-qwen3.8-27b/` + `scripts/download-frank-dragoon-halogen.sh`; 0.1.4 digest still in podman storage |
+| 2026-10-03 → 10-05 | kingjones777 Uncensored `Q6_0_ROCMFPX_AGENT.gguf` 24,856,607,008 B `e966deb3…` + `mtp-Q4_0` `051a1764…` + BF16 mmproj | `kingjones30/ROCmFPX` @ `36e9acd` (`scripts/build-rocmfpx.sh`) | 1 × 262,144 f16 KV, `draft-mtp` n-max 4, vision on | halogen card claimed 31.7 t/s vs 27.6–28.1 measured | `/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/` + `/home/dank/GitPackages/kingjones-ROCmFPX/build` + `download-frank-dragoon-kingjones-uncensored.sh` |
+| 2026-10-02 → 10-03 | aligned `Q8_0_ROCMFPX_AGENT.gguf` 28,214,425,952 B `f3ef2bdb…` | same fork pin | 1 × 262,144 f16 KV, `draft-mtp` 4 (MTP +123 % decode) | the uncensored file was faster *and* uncensored | `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/` + `download-frank-dragoon-kingjones.sh` |
+| 2026-10-01 (one day) | MiMo-V2.6-Distill-Qwen-9B BF16 17.53 GiB + cross-model MTP from `Qwen3.5-9B` | vLLM 0.30 TheRock `kyuz0/vllm-therock-gfx1151` `sha256:98c1f3ac…` | 4 seqs × 262,144 on a scheduler-shared pool, 2 draft tokens | 6.19 t/s with MTP / 9.01 without vs gufo ~15, and no ≥6-bit vLLM quant path boots on gfx1151 | staged `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` until purged; fetch trio in `old/` |
+| 2026-09-28 → 10-01, again 10-01 → 10-02 | Swift-1.5 Uncensored `UD-Q8_K_XL` 31,457,990,272 B `9076f99a…`, then `UD-Q4_K_XL`; z-lab DFlash2 Q4_K_M | gufo v0.1.1 (`b0f8673`), source build | 1 × 262,144, `--draft-policy adaptive --draft-tokens 7`, no KV knob, no Jinja | ROCmFPX gave 8.39 bpw + MTP at 17 t/s | payloads **deleted 2026-10-03** → re-download `download-frank-dragoon-swift.sh` (+ `download-frank-dragoon.sh` for the drafter); tag `frank-dragoon-gufo-q4kxl-20261001`; `scripts/build-gufo.sh` |
+| 2026-09-21 → 09-28 | unsloth `Qwen3.8-27B-GGUF:UD-Q8_K_XL` 31.46 GiB + DFlash2 Q4_K_M 1.14 GiB | unsloth llama.cpp `b11030` prebuilt | 1 × 262,144, q8_0 KV, Qwen-Sharp v22.4.0 template | gufo: +37 % prefill shallow, +80 % at 131k, 4.5× faster load | `/home/dank/GitPackages/unsloth-llama-b11030/` on disk; payload deleted 2026-10-03 → `download-frank-dragoon-unsloth.sh` |
+| 2026-09-21 (same day) | unsloth `Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS` 87.3 GiB in 3 split parts | same prebuilt | 1 × 200,000, MTP depth 2 via explicit `-md` shared Q8_0 head | swapped to the 27B dense hours later | payload + prebuilt on disk at retirement; the parts ARE the payload (concatenating → `invalid split file name`) |
+| 2026-09-21 (same day) | `jcbtc/Qwen3.8-27B-IU4-Kairic-Edge` + 3 `.pfs` = 26.05 GiB | ciru-ai/ROCmFPX `kairic-edge-qwen38-27b-v1.2` @ `205a3e5` | 1 × 262,144, native MTP 4 + n-gram | same-day swap to the 27B dense | payload + runtime intact on disk at retirement |
+| 2026-09-19 → 09-21 | `jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca` | CIRU v4.4.1 (QSA window hotfix) | never written down — that runbook is a frontmatter stub | the stock llama.cpp prebuilt won | `scripts/download-frank-dragoon-orca.sh` |
+| 2026-09-11 → 09-15, again 09-16 → 09-19 | `peonist-ai/halogen-qwen3.8-flash-next` w4b 115.55 GiB + in-place-versioned `overlay.hgn` | halogen-flash-server 0.5.8, rootless Quadlet | 2 × 200,192 over a 400,384 pool, `MAX_TOK 24576`, `HOST_RESERVE 20`, vision OFF | displaced by CIRU Orca; the shape starved co-tenants | re-tag 0.5.8 + first-start `HALOGEN_DOWNLOAD` re-fetch into `/home/dank/models/halogen`; the Quadlet left the repo 2026-10-05, so the unit must be rewritten |
+| 2026-09-15 → 09-16 (one day) | `mradermacher/Qwen3.8-Flash-Next-Uncensored` i1-Q4_K_M 111 GiB + f16 mmproj | ik_llama.cpp `d85a1d7`, 3090 + CPU hybrid | 1 × 262,144, `-ot exps=CPU`, `GGML_CUDA_NO_PINNED_WEIGHTS=1` | 3.8–4.0 t/s: ~100 CPU↔GPU graph splits per token, GPU at 5 % SM | `/home/dank/models/mradermacher-Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`, `/home/dank/GitPackages/ik_llama.cpp/build`, trio `scripts/.AGENTTRASH/frank-dragoon.{env,service}.ik-llama.cpp` + `serve-frank-dragoon.sh.ik-llama.cpp` |
+| 2026-08-30 → 09-11 | kingjones STRIX `Q4_0_ROCMFP4-STRIX.gguf` 113.47 GiB (51.2 B PLE via mmap) | ROCmFPX @ `36e9acd` | 2 × 200,704 over one 401,408 f16 window, spec off, f16 mmproj | halogen-flash-server ≈ 4× prefill | payload dir + `/home/dank/GitPackages/kingjones-ROCmFPX/build`; trio `.AGENTTRASH/frank-dragoon.{env,service}.llama.cpp` + `serve-frank-dragoon.sh.llama.cpp` |
+| 2026-08-25 → 08-29 (restored 08-29 → 08-30 after the Q8_0 AGENT file crashed before it ever served) | `kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP`: `Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf` 23.15 GiB (private ftype 114) + a **SEPARATE** MTP head `mtp-Qwen3.8-27B-Q4_0.gguf` 1.56 GiB via `--model-draft` + `mmproj-Qwen3.8-27B-Q8_0.gguf` 0.59 GiB | CIRU ROCmFPX llama.cpp (`build-promptforge`) | 1 slot at `-c 196608`, `-b 8192 -ub 8192 -ctxcp 32`, `-fa on --no-mmap`, f16 K+V, `draft-mtp` n-max 4 with the head on ROCm1 at f16 KV, QwenSharp template (`enable_thinking`, `preserve_thinking`) behind the `peg-native` + `deepseek` chat/reasoning pairing the Pi agent expected | the Q8_0 AGENT experiment, then the STRIX Q4_0 tier | `/home/dank/models/kingjones-Qwen3.8-27B-STRIX-MTP/` (six card files + projector), runtime `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` |
+| 2026-08-29 → 08-30 (one day) | agentionai FAST-imatrix **`v2/` joined-PLE** file 87.06 GiB | charlie12345/ROCmFPX `build-qwen38` + `scripts/patches/qwen4exp-qsa-f32-mask.patch` | 2 × 200,704 / 401,408 f16 KV, no MTP head loads | kingjones STRIX on its own pinned fork | `/home/dank/GitPackages/ROCmFPX/build-qwen38` + the agentionai payload, both on disk |
+| 2026-08-18 → 08-25 | caf-eemin `Q4_0_ROCMI4.gguf` 13.50 GiB `ad51ecbe…` with embedded NextN/MTP | charlie12345/ROCmFPX, ROCmI4 W4A4 | one window, MTP `n4 / p0.60` | the 27B STRIX tier took the slot | re-fetch into `/home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP/` |
+| through 2026-08-24 | CIRU ActiveFPX PromptForge v3: GGUF 15.44 GiB + 3 `.pfs` + mmproj ≈ 36.7 GiB | CIRU ROCmFPX `build-promptforge` | 1 × 262,144, native MTP depth 4, ngram off | eemin ROCmI4 | `/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/` + `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`; scripts `git revert 4c43839 7aee1df 40d50fe` or checkout `132e147` |
+| 2026-08-22 (same day) | `jcbtc/Qwen3.8-27B-IU4-Kairic-Edge` 15.48 GiB + 3 `.pfs` | ciru-ai/ROCmFPX tag `kairic-edge-qwen38-27b-v1` @ `e97b324` | one window, IU4 W4A4 lane, ubatch 512 | prefill 1.5–2.3× faster but prose decode 15–26 % slower → back to v3 the same day | `/home/dank/models/Kairic-Qwen3.8-27B-IU4-Edge/` + `/home/dank/ciru-qwen38/ROCmFPX-kairic/build-kairic/bin/llama-server` |
+| 2026-08-21 (same day) | unsloth `UD-Q8_K_XL` 31.4 GiB + z-lab DFlash2 Q4_K_M 2.0 GiB | upstream llama.cpp PR #27342, `GGML_VULKAN=ON` | Vulkan0 — HIP has no `TOP_K`, so DFlash could not run there | slower than the CIRU runtime it replaced | both files in `/home/dank/models/`; the PR build must be rebuilt |
+
+Port note: this slot settled on **:7770** on 2026-08-22 (52625 before it); the 2026-08-17 fleet redo
+created it — earlier topologies in [[frank-redo-2026-08-17]].
+
+## Laws learned here
+
+- **KV arithmetic:** ~28 KiB/position incl. block scratch plus ~115 MiB/slot of O(1) state; reservations
+  measured 5.5 GiB @200,704, 7.2 @262,144, 11.0 @401,408, 14.6 @524,288. Quote the `reserving the KV
+  pool` line, not the preflight formula.
+- **One pool, not a partition:** the pool decides how many conversations stay resident; only
+  `HALOGEN_CTX` sets the window, and over-window is a 400 naming the limit, never a truncation. The
+  2026-10-18 doubling to 401,408 was bought as pool parity with [[frank-zealot]] so each slot can
+  hold a maximum-length request at once — it did not lengthen the window by one token.
+- **Budget:** device budget = 121.2 GiB (unrounded `MemTotal` 121.2159) − pinned weights −
+  `HOST_RESERVE_GIB`; need = KV + arena + vision, arena = 3.80 + 8.40 × (`MAX_TOK`/16384) GiB — it
+  predicts every recorded point to 0.1 GiB. `HOST_RESERVE_GIB` is a term in that inequality, **not an
+  allocation**: it cannot return RAM, only take the pool away (ceiling 16 at the 524,288 shape, ≥18
+  clamped it). Free RAM by cutting the pool or `MAX_TOK`.
+- **`HALOGEN_KV_POOL_FIT` prints its sizing block only when it clamped** — silence is not headroom and a
+  green `/health` is not the pool you asked for.
+- **RAM the pool takes is RAM the 47.68 GiB n-gram table loses** (page cache, `Pin::None`). The real
+  failure is file-cache starvation, not RAM pressure — long prompts crawl with the disk busy and the
+  process uninterruptible, and the fix is a *smaller* pool.
+- Pinned weights count as reclaimable file cache, so `free`, `MemAvailable` and Grafana overstate free
+  RAM by ~63 GiB; believe the engine's `… in all` line.
+- **Pin gate:** the engine refuses to pin below checkpoint + 16 GiB of `MemAvailable` (pinned pages
+  cannot swap, and under the floor the n-gram table pages in from disk every long prompt), so a
+  RAM-short host crash-loops the unit every ~5 s per boot — 2026-10-06: 8 boots at `MemAvailable`
+  74.77 GiB against the 78.11 GiB needed, self-resolved in ~2 min when the dying 27B engine freed RAM.
+  The serve script waits `FRANK_DRAGOON_RAM_WAIT` (600 s) and refuses loudly if the RAM never clears.
+- `HALOGEN_FLASH_PIN_TRUNK=0` is refused by the v2 loader (and cost several times decode on the w4b
+  build): disk-backing the weights is not a lever here.
+- **MTP is always on:** `HALOGEN_SPEC_ADAPT=0` is mandatory — the adaptive default disables the head for
+  whole requests and **halves decode**. Depth 2 all-round, 3 for code.
+- **Batch XOR speculation was engine-version-specific:** halogen 0.1.4 lost the drafter above one slot
+  (41 → 14 t/s); flash-server runs two slots *with* the head. On the ROCmFPX/llama.cpp line
+  `--spec-draft-n-max` defaults to **16** and must be pinned to 4 — the measured knee on the dense
+  family, with the draft head offloaded (`--spec-draft-ngl 999`) and ngram never layered under it.
+- **Prompt cache is free here and was manual on the last engine:** flash-server ships it on and its
+  auto-sizing fits this box's ledger (no pin) — a 100k follow-up re-prefills in ~2 s instead of ~88 s
+  cold (env, 2026-10-17), and the 2026-10-18 A/B measured 81.5 s cold against 0.3 s warm at 101,987
+  prompt tokens. halogen 0.1.4's auto-sizing refused here and re-prefilled every 79–81k turn in
+  85–180 s until `HALOGEN_CACHE_MB=16384` was pinned. What the pool takes is what the table loses, so
+  re-measure the warm ratio after any pool change.
+- **400-zone law:** the harness window MUST sit ~45k under the engine ceiling (`contextWindow:
+  155648` = 200,704 − 45,056) because the harness clamps `max_tokens` to `contextWindow − est − 4096`,
+  its chars/4 `est` under-counts code/agent traffic by ~3 %, and compaction only fires at 85 %. The
+  3 % is measured, not theory: 2026-10-07 at the old 200,192 ceiling, est 152,938 vs server 157,388 —
+  over by 354, a 400. At the live shape compaction fires at 132,301 est ≈ 136k measured (the 180k
+  target of the 262,144 era is unreachable — the window itself is 155,648), and the worst-case request
+  is 155,648 − 4,096 + 3 % ≈ 156.2k ≤ 200,704: ~9.5× the error of margin, robust to a ~25 %
+  under-count. Compaction rides [[frank-zealot]] at ≈ threshold − keepRecent(20k) ≈ 112k input.
+- **Admission reserves prompt + `max_tokens`:** an un-fittable request waits in arrival order, then
+  503s `engine_busy` for the whole queue after `QUEUE_TIMEOUT` — and the cap/timeout pair moves
+  together: 65536 tokens at ~10 t/s serial needs >6550 s, which is why 7200 is pinned (env, 2026-10-17).
+- **Depth is the cost on this silicon:** ~8 µs per token of context per round (KV reads ~8 GB/s against
+  180–240 GB/s of weight streaming) — 30–50 t/s shallow, 3.3–3.8 at 79–81k, so deep-conversation roles
+  belong on the fp8-KV endpoint [[frank-archon]]. BF16 dense decode tops out at 13.1 t/s (≈94 % of the
+  roof): only speculative decode moves it.
+- **The file picks the runtime, never the reverse:** v2 `.hgn` only in flash-server, the 27B `.hgn` only
+  in halogen, ROCmFPX types 101/102/103/105/108 in no stock loader; fork builds pin by SHA —
+  pre-`36e9acd` ROCmFPX loads the AGENT files then SIGSEGVs in `ggml_compute_forward_set_rows`.
+- **Same filename ≠ same model:** native v2 is 66,687,678,432 B, the Swift1.5 r2 is 66,687,678,464 B.
+  Byte count + LFS sha256 is the only discriminator, and upstream versions sidecars in place, so equal
+  sizes prove nothing.
+- **Companion law:** the n-gram table is REQUIRED beside the checkpoint; if the publisher ships only the
+  main file, hardlink the engine vendor's stock table/tower/tokenizer (a hardlink is a regular file and
+  passes the no-symlink gate) and referee with the engine's own tools — a mismatched table raises KL
+  *everywhere* and degrades retrieval, which is what NIAH + needle probes rule out.
+- **Payload gates:** flat tokenizer directory, every entry a regular file (HF cache symlinks dangle
+  inside the mount), and the serve gate must mirror the deployer exactly or deploy goes green while the
+  unit crash-loops.
+- Serve-script lore (this file has been bitten by all three): capture `podman images -q` into a
+  variable *before* grepping it — under `pipefail`, grep's early exit killed the pipeline and the
+  deploy refused with "no match for the pinned digest" on a box that had the image (2026-09-30); never
+  put a comment inside the `podman run` backslash chain — a mid-chain `#` terminates the continuation
+  and runc gets `--name run` (2026-10-05, exit-127 crash loop); pass only `--ulimit memlock`, since
+  Dankclaw's nofile hard limit is 524,288 and rootless runc cannot raise it (exit 126), and clear the
+  legacy `frank-dragoon` container name before starting.
+- **Bench discipline (this endpoint wrote the fleet law):** stop the service for any GPU probe
+  (`census-frank-dragoon.sh verify` is the only GPU-free one); `gufo bench` cannot rank speculative
+  settings and inverts them; ≥128 generated tokens or the per-token figure is first-token noise; card
+  acceptance/t-s numbers are file properties, not promises — the live card's 53–57 t/s greedy decode
+  and ~1,640 t/s prefill at 34k measured 50.0 and 1,604 here (2026-10-17), and the 27B card's 31.7
+  t/s measured 27.6–28.1; health 200 does not prove your pid owns the port; there are no scratch ports
+  on this host — A/B through `scripts/trial-frank-dragoon.sh` drop-ins; never `pkill -f llama-server`
+  (frank-zealot is one), sweep by payload name.
+- **Dead ends — do not re-try:** ngram under MTP or DFlash (slower at every depth; on recurrent
+  Gated-DeltaNet layers it also disables MTP's rollback path, −7…−15 %); DFlash across parallel slots
+  (shared draft ctx spuriously emits EOG); `-ot`/`-ncmoe` on a hybrid split (disables mmap, page-locks
+  ~108 GiB and stranded the host until a power cycle); `--no-mmap` on any PLE-table file (silent cgroup
+  OOM-kill); forcing PLE to CPU (23 → 13.4 t/s); `HALOGEN_FA_TQ=256` (no decode win, −9.5 % prefill);
+  big ubatches as a decode knob (prefill-only, ub 512 fastest); Flash-Next MTP heads on the qwen4exp
+  loader (only `blk.48`, never the full per-layer norm set); mradermacher i1 GGUFs (NextN stripped);
+  vLLM quant paths on gfx1151; a custom Jinja template on gufo (no Jinja engine — send the `terse` text
+  as a client system message instead).
+- Build lore if a fork is rebuilt: purge Ubuntu `librocwmma-dev` (shadows AMD's headers, `-I` cannot beat
+  it), pin `--gcc-install-dir` to GCC 15, `-DLLAMA_BUILD_WEBUI=OFF`; four HIP processes is the device
+  ceiling and ComfyUI is a co-tenant; `amdgpu: SVM mapping failed` spam is a symptom, not a cause.
+
+## Contract (wire)
+
+- Live: `/v1/models`, `/health`, `/metrics`, POST `/v1/chat/completions`; `404`: `/ping`, `/props`,
+  `/completion`, `/tokenize`. **No `Server-Timing` header and no `/completion` timing object** — measure
+  wall-clock plus `usage.prompt_tokens`.
+- Thinking lands in `reasoning_content` and **counts against `max_tokens`**: a budget that dies
+  mid-thought returns `finish_reason: "length"` with empty `content`, so clients read both fields; size
+  vision probes at ≥512 tokens or the model looks blind. Budget for the base reasoning long: 1,476
+  thinking tokens mean on the 2026-10-17 battery, against 784 for the retired Swift1.5 finetune (×1.9)
+  and 905 for the censored 27B.
+- `reasoning_effort` accepts only `xhigh|medium|low` (anything else is a template error — the
+  `models.yml` fold stays) and there is **no `"none"`**: disable thinking with `enable_thinking:false`,
+  which is also what the llama.cpp-era QwenSharp template wanted (`reasoning_effort:"none"` threw
+  there too). `preserve_reasoning` is a 400 on this engine; the llama.cpp-era key was
+  `chat_template_kwargs.preserve_thinking`, which decided whether `<think>` blocks stayed in recorded
+  assistant messages. Sampler defaults 1.0/0.95/20 are server-side, so an absent temperature is NOT
+  greedy (halogen 0.1.4 and w4b decoded greedily); unknown sampling fields are a 400, out-of-range
+  values are refused not clamped, and `response_format` is unsupported.
+- Over-ceiling prompt → 400 naming the exact overage, rejected by preflight in <1 s, so probing the
+  ceiling is free; over-cap `max_tokens` → 400 naming the cap; `max_tokens`, `max_completion_tokens` and
+  `max_output_tokens` all work and must agree.
+- Vision: OpenAI `image_url` parts only (top-level `images` is the frank-zealot dialect), ≥256 px or the
+  model answers "video", ~2,040 tokens per image, `http(s)` image URLs refused on purpose.
+- The first request after a boot is a warmup artifact and may sit silent 45–60 s with 503
+  `engine_unresponsive` then self-recover — do not restart over it; `/health` 503 during load is loading.
+  Flags read once at startup, all bitwise except `HALOGEN_CK_OVERLAY`, `HALOGEN_MATMUL_TUNING_FILE`,
+  `HALOGEN_INDEXER_BUDGET`, `HALOGEN_COMPOSABLE_CONTEXT`.
+- Deploy waits for :7770 to be free: a rootless container binds it at CREATE and exits 126, and a
+  leftover Quadlet beside the root unit guarantees the race.
+
+## Files
+
+- `scripts/frank-dragoon.env` · `scripts/serve-frank-dragoon.sh` (gates, RAM pin-gate wait, foreground
+  `podman run`, ledger to `/home/dank/logs/frank_dragoon.log`) · `scripts/frank-dragoon.service`
+  (`User=dank`, `LimitMEMLOCK=infinity`, `Restart=always`, `OOMScoreAdjust=-500`, `ExecStop=podman stop
+  -t 60`) → `deploy_llm_dragoon` in `scripts/deploy-remote.sh`.
+- Fetchers: `download-frank-dragoon-v2abl.sh` (live); era rollbacks `-flashnext`, `-halogen`,
+  `-kingjones`, `-kingjones-uncensored`, `-swift`, `-unsloth`, `-q4`, `-orca`,
+  `download-frank-dragoon.sh`; runtime pins `build-rocmfpx.sh`, `build-gufo.sh`.
+- Verification: `tests/frank_dragoon_unit.bats` (payload gates, deployer/serve parity, spec block, KV
+  band), `tests/llm_units.bats`, `tests/llm_secrets.bats`, `scripts/census-frank-dragoon.sh`,
+  `scripts/battery-frank-dragoon.py`, `scripts/stage-frank-dragoon-eval.sh`,
+  `scripts/trial-frank-dragoon.sh`.
+- Related: [[deployment]] · [[Model Swap and Rollback]] · [[frank-zealot]] · [[frank-archon]] ·
+  [[dankclaw-iommu-translation-off]] · [[frank-redo-2026-08-17]] · [[Dankclaw System Specs]] ·
+  [[DFlash DSpark Speculative Decoding]] · [[qwen3.8-spec-matrix]] · [[qwen3.8-strix-optimizations]].

serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md

@@ -211,7 +211,7 @@ re-checking that 35:30 and 37:28 still fail so the gate keeps discriminating.
 
 - [[frank-medium-llamacpp-dflash2]] — the deployment this tuned, and its split
   history
-- [[frank-zealot-benchmark]] — the same sweep on a CUDA 3090, where a 16k prompt
+- [[frank-zealot]] — the same sweep on a CUDA 3090, where a 16k prompt
   at depth 0 made prefill climb all the way to ubatch 2048. Depth is the
   difference: with no KV to attend over, per-pass overhead dominates and a
   bigger micro-batch wins; at 64k of resident KV it does not.

serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md

@@ -724,7 +724,7 @@ fleet.
   2026-09-15, whose `--split-mode layer` measurements this service reuses
 - [[Dual-R9700-Cooperation]] — why the two cards cannot take part in a
   collective, and every software escape that is closed
-- [[frank-zealot-qwen38-gsq-rco]] — the fleet's other llama.cpp endpoint (RTX
+- [[frank-zealot]] — the fleet's other llama.cpp endpoint (RTX
   3090, MTP, the image path)
 - [[Dankserver System Specs]] — hardware inventory
 - [[frank-medium-batch-ubatch]] — the batch/ubatch sweep that set the deployed

serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md

@@ -25,7 +25,7 @@ timestamp: 2026-09-15
 when the RTX 3090 left the box and two **AMD Radeon AI PRO R9700** cards
 (Navi 48, RDNA4, `gfx1201`, 32,624 MiB each, PCIe 5.0 x16) took its place.
 It replaced the retired FreeToken/CUDA `frank-small`
-([[freetoken-frank-small]]), which cannot run on this hardware at all.
+([[frank-zealot]]), which cannot run on this hardware at all.
 
 Stack: **charlie12345/ROCmFPX llama.cpp** (`main`, `c49ebdb`, 2026-08-22)
 serving **kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF** with the
@@ -213,7 +213,7 @@ depend on any of it.
 
 ## See Also
 
-- [[freetoken-frank-small]] — the retired predecessor (FreeToken/CUDA on the 3090)
+- [[frank-zealot]] — the retired predecessor (FreeToken/CUDA on the 3090)
 - [[kingjones-q8agent-rocmfpx]] — the gfx1151 investigation of this checkpoint family
 - [[Llama Cpp Runtimes]] — where the ROCmFPX fork sits among the runtimes
 - [[Dankserver System Specs]] — current hardware

serverAppConfigs-docs/LLM/frank-npu.md

@@ -25,7 +25,7 @@ timestamp: 2026-09-11
 > file and serve script are staged in `serverAppConfigs/.AGENTTRASH/`
 > (suffixed `.flm-npu`), and the host unit is still masked
 > (`systemctl is-enabled frank-npu` → `masked`, read 2026-10-18). Port 7771
-> belongs to [[frank-zealot-llama-prism-bonsai2-pq20]] today.
+> belongs to [[frank-zealot]] today.
 
 The third inference endpoint on Dankclaw: a FastFlowLM (FLM) v1.0.1 server
 (`/usr/bin/flm`, dpkg `fastflowlm`) on the **XDNA2 NPU** (50 TOPS), serving the
@@ -161,4 +161,4 @@ retirement; the routes are mapped in the table above.
 
 - [FastFlowLM](https://github.com/FastFlowLM/FastFlowLM) — FLM runtime and NPU model packs
 - [[dankclaw-iommu-translation-off]] — the decision that retired this endpoint
-- [[halogen-flash-server]] — the surviving image-input path
+- [[frank-dragoon]] — the surviving image-input path

serverAppConfigs-docs/LLM/frank-redo-2026-08-17.md

@@ -14,11 +14,11 @@ The pipeline-parallel split and the FastFlowLM NPU endpoint were retired.
 The old scripts moved to `old/scripts/` in this repository.
 
 > **HISTORY — the topology below is two fleet generations dead.** Today the
-> fleet is three endpoints: [[frank-archon-radiance-qwen38-iq4r]]
+> fleet is three endpoints: [[frank-archon]]
 > (`192.168.0.235:7770`, radiance 1.0.13, 2×R9700 TP=2),
-> [[frank-dragoon-flash-server-v2-native]] (`192.168.0.143:7770`, halogen
+> [[frank-dragoon]] (`192.168.0.143:7770`, halogen
 > flash-server 0.16.2 on the iGPU) and
-> [[frank-zealot-llama-prism-bonsai2-pq20]] (`192.168.0.143:7771`, llama.cpp
+> [[frank-zealot]] (`192.168.0.143:7771`, llama.cpp
 > PrismML fork build 10754 on the RTX 3090 — the card that sat in Dankserver
 > until 2026-09-15). No vLLM and no ROCm endpoint serves anything; there is no
 > pipeline parallelism and no NPU endpoint.

serverAppConfigs-docs/LLM/frank-zealot-benchmark.md

@@ -1,193 +0,0 @@
----
-timestamp: 2026-09-03
-type: reference
-title: frank-zealot Benchmark — ncmoe and Batch Tuning
-description: llama-bench sweep of MoE CPU offload (ncmoe) and batch/ubatch on the RTX 3090 for Qwen3.6-35B-A3B Q6_K_P.
-tags:
-  - benchmark
-  - llamacpp
-  - ncmoe
-  - ubatch
-  - dankserver
----
-
-# frank-zealot Benchmark — ncmoe and Batch/Ubatch Tuning
-
-> **HISTORY (runs 2026-08-13 and 2026-08-25, Dankserver's RTX 3090).** That
-> card was pulled from Dankserver on 2026-09-15 (`lspci | grep -ci nvidia` → 0
-> there on 2026-10-18 — [[Dankserver System Specs]]) and today drives
-> frank-zealot on **Dankclaw**: [[frank-zealot-llama-prism-bonsai2-pq20]]
-> (`Ternary-Bonsai-2-27B-PQ2_0.gguf`, 2 × 200,704 q8_0 KV, MTP off, no
-> `--n-cpu-moe` — the model fits the card outright). What survived the payloads:
-> decode is ubatch-invariant, CUDA prefill climbs monotonically to ubatch 2048,
-> and the logical `--batch-size` is neutral — the live env still ships
-> 4096/2048 (`scripts/frank-zealot.env:44-45`).
-
-This page records a llama-bench sweep on Dankserver (RTX 3090, 24 GiB) for
-the HauhauCS Qwen3.6-35B-A3B Q6_K_P model. The sweep decided the final
-`--n-cpu-moe` and `--ubatch-size` values for the llama.cpp rung then served on
-that card (the `frank-zealot` name dates from 2026-10-06/10-09).
-
-Run date: 2026-08-13. Build: llama.cpp `555881ebc` (10121). CUDA, 7 threads,
-`--cache-type-k q8_0 --cache-type-v f16`, `--fit-target 0` (KV auto-sized to
-fill VRAM). Prompt 16384 tokens, generation 512 tokens, 2 repetitions.
-
-## Headline Findings
-
-### Generation does not depend on batch or ubatch
-
-For every `ncmoe` value, generation throughput stayed flat across all
-`ubatch` values from 256 to 4096. The delta between the lowest and highest
-was under one token per second. Generation depends only on how many MoE
-expert layers stay on the GPU as `ncmoe` gets smaller.
-
-| `ncmoe` | Expert layers on GPU | Generation (t/s) |
-| ------- | -------------------- | ---------------- |
-| 40      | 0                    | 40.9             |
-| 38      | 2                    | 42.6             |
-| 36      | 4                    | 44.4             |
-| 35      | 5                    | 45.5             |
-| 34      | 6                    | 46.4             |
-| 32      | 8                    | 48.7             |
-| 30      | 10                   | 51.1             |
-
-Each pair of expert layers moved to the GPU adds roughly 1.1-2.0 t/s of
-generation. Moving from `ncmoe 35` to `ncmoe 32` adds about 3.2 t/s, or 7
-percent.
-
-### Prefill and graph memory scale with ubatch
-
-Prefill throughput climbs with `ubatch` because a prompt splits into
-`prompt / ubatch` micro-batches. For `ncmoe 35` at 16384 prompt tokens:
-
-| `ubatch` | Prefill (t/s) | Peak graph VRAM (MiB) |
-| -------- | ------------- | --------------------- |
-| 256      | 245           | 6838                  |
-| 512      | 416           | 6852                  |
-| 768      | 556           | 7074                  |
-| 1024     | 710           | 7330                  |
-| 1536     | 895           | 7846                  |
-| 2048     | 1188          | 8574                  |
-
-The logical `--batch-size` does not matter. At a fixed `ubatch`, `batch`
-values of 2048, 4096, and 8192 gave identical prefill speed and peak VRAM.
-
-## Peak VRAM by ncmoe and ubatch
-
-Peak VRAM (llama-bench, single context) rises both when `ncmoe` drops
-(more experts on GPU) and when `ubatch` rises (bigger compute graph):
-
-| `ncmoe` | ub 512 | ub 1024 | ub 1536 | ub 2048 |
-| ------- | ------ | ------- | ------- | ------- |
-| 38      | 4590   | 5068    | 5584    | 6086    |
-| 36      | 6098   | 6576    | 7092    | 7594    |
-| 35      | 6852   | 7330    | 7846    | 8574    |
-| 34      | 7606   | 8084    | 8600    | 9102    |
-| 32      | 9114   | 9592    | 10108   | 10610   |
-| 30      | 10622  | 11100   | 11616   | 12118   |
-
-## VRAM Budget Model
-
-The live service holds a fixed 700416-token KV pool (f16/f16 since
-2026-08-14; refitted from 800000) plus the mmproj vision tower plus CUDA
-overhead. The model below fits llama-bench peak VRAM into a 24 GiB
-budget:
-
-```
-service VRAM = llama-bench peak + KV pool (13.68 GiB f16) + overhead (2.2 GiB)
-```
-
-This model reproduces the historical OOMs. Configurations whose sum
-exceeds 24 GiB fail. For example, `ncmoe 32 - ubatch 2048` sums to about
-24.4 GiB and fails, which matches the recorded OOMs at `ncmoe 32` and
-`33`. The f16/f16 KV pool (2026-08-14) costs 20,480 bytes per cell; at
-800768 cells it is 15,640 MiB and does not fit the 24 GiB card, so the
-service runs 700416 cells (13,680 MiB).
-
-## Candidate Configurations
-
-| Configuration      | Gen (t/s) | Prefill (t/s) | Headroom (MiB) | Fit |
-| ------------------ | --------- | ------------- | -------------- | --- |
-| ncmoe 35, ub 2048  | 45.5      | 1188          | 2035           | OK  |
-| ncmoe 34, ub 2048  | 46.4      | 1221          | 1507           | OK  |
-| ncmoe 34, ub 1536  | 46.5      | 914           | 2009           | OK  |
-| ncmoe 32, ub 1024  | 48.7      | 766           | 1017           | OK  |
-| ncmoe 32, ub 768   | 48.8      | 600           | 1273           | OK  |
-| ncmoe 30, ub 1536  | 51.2      | 1009          | under          | OOM |
-
-## Chosen State
-
-> **Live drift (2026-08-22 audit):** the deployed `frank-zealot` now runs `--n-cpu-moe 38`, `--batch-size 4096`, `--ubatch-size 2048` (ncmoe raised from 36 for stability; batch lowered). The values below are the original benchmark recommendation.
-
-Prefill speed is the priority, so the state keeps the top prefill tier and
-moves one expert layer to CPU for stability:
-
-```
---n-cpu-moe 36     # 4 expert layers on GPU
---batch-size   8192
---ubatch-size  3072
-```
-
-This gives prefill about 1427 t/s at 16384 prompt tokens, generation about
-44.4 t/s, and about 2.0 GiB of headroom at the 700416-token f16/f16 pool
-(2026-08-14 refit from 800000; the dual-f16 pool costs 13.68 GiB and
-removes the context-proportional flash-attn F16-conversion workspace that
-caused the cuBLAS crash class).
-
-`ncmoe 36` came from a live crash rather than this sweep. At `ncmoe 35` with
-`ubatch 3072`, a big prefill burst failed a cuBLAS call under memory
-pressure (a `cudaErrorInvalidValue` in `ggml_cuda_mul_mat_cublas`), which
-killed the service. Raising `ncmoe` one step frees about 0.75 GiB of peak
-VRAM (the grid shows `ncmoe 36 - ubatch 3072` peak 8606 MiB versus 9360 MiB
-at `ncmoe 35`) for about 2 percent less prefill and 2 percent less decode.
-
-If a crash still occurs under load, set `GGML_CUDA_FORCE_MMQ=1` to bypass
-the cuBLAS path for quantized matmuls, or drop `ubatch`, or shrink the
-context pool.
-
-If decode speed matters more than prefill, swap to `ncmoe 32 - ubatch 1024`
-(generation about 48.7 t/s, prefill about 766 t/s, about 1.0 GiB headroom).
-
-## Ornith Q6_K — 8K `ubatch` Sweep Without Speculation
-
-Run date: 2026-08-25. This controlled sweep used the active Ornith Q6_K model
-and hybrid expert placement, but explicitly disabled speculative decoding with
-`--spec-type none`. It used one slot, an 8192-token context, a fixed logical
-`--batch-size 4096`, prompt caching disabled, and one unique approximately
-8K-token prompt per point. The production service was stopped during the
-sweep and automatically restarted afterward.
-
-| `ubatch` | Prompt tokens/s | Generation tokens/s | CUDA compute reservation |
-|---:|---:|---:|---:|
-| 1024 | 1944 | 53.91 | 248 MiB |
-| 1280 | 2030 | 53.87 | 248 MiB |
-| 1536 | 2190 | 53.66 | 248 MiB |
-| 1792 | 2203 | 53.69 | 248 MiB |
-| 2048 | 2423 | 53.80 | 248 MiB |
-
-Prefill improves through `ubatch=2048`, while generation is effectively flat.
-The 8K prompt sweep therefore favors `ubatch=2048` as the balanced point; a
-larger value was not tested in this corrected no-spec run.
-
-The same workload with `draft-mtp` enabled produced approximately 26.8–35.2
-generation tokens/s at these `ubatch` values, with variable draft acceptance.
-The no-spec path produced approximately 53.7–53.9 generation tokens/s. This
-means MTP was a net throughput loss for this workload and configuration,
-despite accepting draft tokens. The production service remains on MTP until a
-separate decision is made; acceptance count alone is not sufficient to select
-speculation.
-
-## Limitations
-
-llama-bench measures raw decode and prefill. It does not run speculative
-decoding, parallel slots, or the mmproj vision tower. The live service adds
-those on top. The VRAM model uses a fixed 2.2 GiB overhead estimate; the
-real overhead varies a little with load. The Ornith sweep above used the
-OpenAI-compatible server API rather than `llama-bench`, and its throughput
-includes the server's timing instrumentation.
-
-## See Also
-
-- [[Dankserver System Specs]]
-- [[deployment]]
-- [[DFlash DSpark Speculative Decoding]]

serverAppConfigs-docs/LLM/frank-zealot-gemma4-12b.md

@@ -1,134 +0,0 @@
----
-type: reference
-title: "frank-zealot on Dankclaw — llama.cpp, Gemma4-12B QAT Q4_K_M"
-description: "The RETIRED frank-zealot runbook (2026-09-17 → 10-01) — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves."
-tags:
-  - llm
-  - frank-zealot
-  - dankclaw
-  - llama.cpp
-  - rtx3090
-  - gemma
-  - mtp
-  - vision
-timestamp: 2026-09-17
----
-
-# frank-zealot — llama.cpp Gemma4-12B QAT Q4_K_M (MTP + ngram-mod)
-
-> **RETIRED 2026-09-17 → 2026-10-01** (AGENTS.md Runbook Index). It replaced
-> [[frank-zealot-qwen38-gsq-rco]] (ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S) and
-> [[frank-zealot-vllm-gemma4]] (the same-day vLLM experiment).
-> Same port (7771), same alias (`frank-small` — this endpoint only carried the
-> `frank-zealot` name from 2026-10-06/10-09), same client contract surface.
-> :7771 today runs [[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML
-> fork build 10754, `Ternary-Bonsai-2-27B-PQ2_0.gguf`, 2 × 200,704 q8_0 KV, MTP
-> off); the audio input this page documents left the fleet with this payload.
-
-## Identity
-
-- **Endpoint:** `frank-small` at `192.168.0.143:7771` (Dankclaw) —
-  `git show e23df8b:scripts/frank-small.env` → `ALIAS=frank-small`.
-- **Runtime:** mainline llama.cpp master `37b53fd`, CUDA build for sm_86 with
-  `-DGGML_CUDA_FA_ALL_QUANTS=ON`
-  (`/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server`).
-  The FA_ALL_QUANTS flag is required: without it, q8_0 KV + flash attention
-  collapses prefill ~12x on CUDA. Host check 2026-10-18: that binary reports
-  `0.4.1-dev (build 1, commit 37b53fd)` and its `CMakeCache.txt` reads
-  `GGML_CUDA_FA_ALL_QUANTS:BOOL=ON`.
-- **Service shape:** root systemd unit `scripts/frank-zealot.service`
-  (`User=dank`), launched by `scripts/serve-frank-zealot.sh` with tunables from
-  `scripts/frank-zealot.env`.
-
-## Shape
-
-- **Model:** `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` at
-  Q4_K_M (6.87 GiB). Architecture `gemma4`: 48 layers, MHA with 16 kv heads
-  x 512 head_dim, rope freq_base 1e6, final logit softcapping 30, per-layer
-  token embeddings (PLE) inside the file, and a sliding-window attention
-  pattern (`attention.sliding_window = 1024`) — most layers cache only a
-  1024-position window, which is why the KV pool is a fraction of the
-  full-ctx x all-layers number.
-- **Context:** four FULL 262,144-token slots (`CTX_SIZE=1048576`,
-  `PARALLEL=4`). The explicit `CTX_SIZE` bypasses `--fit` on purpose: `--fit`
-  maximises the TOTAL pool, which would shrink each slot below the full window
-  once parallel > 1. The 4-slot boot idles at 22,346 MiB; each extra
-  full-context slot costs 3,450 MiB, and a fifth projects to ~25.8 GiB — past
-  the card. 4 is the measured max.
-- **Placement:** the whole model in VRAM (`--gpu-layers 999`); the host CPU
-  only tokenizes and samples (its CPU belongs to the halogen engine serving
-  frank-dragoon).
-
-## Spec method
-
-The MTP head ships as a **separate** file (`mtp-gemma-4-12B-it.gguf`, arch
-`gemma4-assistant`, 4 nextn layers, 0.24 GiB), loaded via `--model-draft`
-with `--spec-type draft-mtp`. The same `--spec-type` value chains the
-draftless **ngram-mod** matcher, which attaches unconditionally; two
-context-backed drafters cannot share one process, so `draft-mtp,ngram-mod` is
-the only legal shape that carries both. Draft depth 3 + poll 50 (measured:
-depth 4 + poll 100 decoded WORSE at long context — the longer verification
-round costs more attention re-read than the extra accepted token earns).
-
-## KV
-
-**q8_0** (`--cache-type-k q8_0 --cache-type-v q8_0`), with flash attention
-(`--flash-attn on`, required for quantized KV). CPU-side cache capped at 2048
-MiB (`CACHE_RAM=2048`) to give back ~6 GiB of LPDDR — Gemma4 runs fully on the
-3090, so the CPU cache is not the hot path, and it is not a DeltaNet model so
-the >12.8 GiB floor does not apply.
-
-## Vision + audio
-
-`--mmproj` loads the BF16 projector
-(`mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf`, 11 tensors,
-52M params), which also carries an audio encoder. The fleet's image path
-lives here. Audio works: `input_audio` content parts transcribe verbatim
-(verified 2026-09-17 with a 16k mono WAV). Fleet-wide audio ended with this
-payload: no fleet endpoint takes audio input today.
-
-## Payload
-
-The main GGUF, the BF16 mmproj, and the MTP sidecar lived under
-`/home/dank/models/Gemma4-12B-QAT`; they were staged to
-`.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/` on 2026-10-02 and
-that stage dir is gone as of 2026-10-18, so the payload needs a re-fetch. The serve script refuses to start if any
-is missing (a missing file is an unfinished download, and the unit's restart
-policy would otherwise turn it into a crash loop that reads like a runtime
-fault).
-
-## Client contract
-
-- **Alias:** `frank-small` (the `id` at `/v1/models`).
-- **Reasoning:** extracted to `reasoning_content` (`--reasoning-format
-  deepseek`, unlimited budget).
-- **Tools:** supported (`--jinja` is pinned even though this build defaults
-  it on, so a future build that flips the default cannot silently take tools
-  away).
-- **Vision + audio:** both supported in this era; audio is gone fleet-wide
-  today (`input: [text, image]` — `~/.omp/agent/models.yml:214`).
-- **Sampling:** greedy by default (fleet contract); a client that sends a
-  temperature gets its own, and the harness applies per-model overrides from
-  `models.yml`.
-
-## Measured (max length)
-
-At 254k-token prefill / 253k-ctx decode, 4 slots, q8_0 KV, ubatch 1024:
-pp ~965 t/s, tg 257–302 t/s, idle 23,300 MiB of 24,576. Larger ubatches
-(2048) OOM the MTP draft context's pp compute buffer and must not be tried.
-
-## Rollback
-
-Payload and runtime for [[frank-zealot-qwen38-gsq-rco]] (ISTA-DASLab
-Qwen3.8-27B-GSQ-RCO IQ3_S) are **not on disk any more** (the 2026-10-18 census
-of `/home/dank/models/` has no GSQ-RCO directory and the era fetch script is
-not in the tree) — that rollback re-downloads from ISTA-DASLab.
-[[frank-zealot-vllm-gemma4]] is the same-day vLLM experiment; its weights are
-gone too, though its derived image is still in rootless storage
-(`localhost/vllm/vllm-openai:frank-small`, id `8dad46d79c46`).
-
-# Citations
-
-- Model + mmproj + MTP sidecar: https://huggingface.co/HauhauCS
-- llama.cpp: https://github.com/ggml-org/llama.cpp
-- Serve script: `scripts/serve-frank-zealot.sh`; tunables: `scripts/frank-zealot.env`

serverAppConfigs-docs/LLM/frank-zealot-llama-gemma4-q4km-200704.md

@@ -1,116 +0,0 @@
-# frank-zealot — llama.cpp CUDA, Gemma4-26B-A4B Q4_K_M, 1×200,704 (Dankclaw)
-
-**RETIRED 2026-10-07** — same day it went live (AGENTS.md Runbook Index).
-Superseded by
-[[frank-zealot-llama-qwen35-9b-q6k-2x200704]] (Qwen3.5-9B Q6_K at
-2 × 200,704 with exact KV); :7771 today runs
-[[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML fork build 10754,
-`Ternary-Bonsai-2-27B-PQ2_0.gguf`, 2 × 200,704 q8_0 KV, MTP off). The payload
-stays at `/home/dank/models/Gemma4-26B-A4B/` as the rollback (all three files
-present in the 2026-10-18 host census). This page keeps the
-Gemma4 KV-billing law (the `attention_k_eq_v` pair-billing measurement) and
-the drafter-cost boots, both of which outlive the model.
-
-## Payload
-
-|File|Path|Bytes|
-|---|---|---|
-|Main (Q4_K_M)|`/home/dank/models/Gemma4-26B-A4B/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf`|16,796,015,520 on disk; the engine reports 16,780,192,888|
-|Vision (mmproj BF16)|`/home/dank/models/Gemma4-26B-A4B/mmproj-Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf`|1,194,827,776 B on disk (2026-10-18) = 1.11 GiB|
-|MTP assistant (on disk, NOT loaded)|`/home/dank/models/Gemma4-26B-A4B/mtp-gemma-4-26B-A4B-it.gguf`|251,937,728 B on disk (2026-10-18) = 0.23 GiB|
-
-Repo: `HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP`
-(base `google/gemma-4-26B-A4B-it`). Downloaded 2026-10-07 by plain
-`curl -fL` (no token needed, ungated).
-
-## Shape
-
-```
---ctx-size 200704 --parallel 1        # ONE conversation, Frank's 200k cap
---flash-attn on
---cache-type-k f16 --cache-type-v q8_0   # exact K, FP8-class V (q8_0)
---batch-size 4096 --ubatch-size 1024
---n-gpu-layers 99
---log-verbosity 4                     # debug: KV layout, buffers, graph capture
---spec-type none                      # MTP OFF (see below)
---mmproj <BF16 projector>
---jinja --reasoning auto --reasoning-format deepseek
-```
-
-Measured boot at **q8_0 K + q8_0 V, ubatch 2048, MTP off** (2026-10-07
-morning): **21,890 MiB used / 2,235 MiB free** (`nvidia-smi`). With MTP ON
-(the assistant loaded) the same shape was **23,806 / 319** — the drafter's
-context costs ~1.9 GiB because its KV is built against the target's full
-window ([[Llama Cpp Serving Sizing]]).
-
-**Deployed shape (2026-10-07 evening, Frank): K = f16, ubatch 1024, verbosity
-4. Measured boot: 21,868 MiB used / 2,257 MiB free** — i.e. the f16-K raise
-(+2,161 MiB of KV, see the accounting below) was paid for by the ubatch cut
-(2048 → 1024 returns ~616 MiB of compute buffer) plus slot-geometry variance.
-Exact K came essentially free. The verbosity-4 ledger (journal, `llama_kv_cache`
-lines): model 16,002.91 MiB (CUDA0) + 577.50 MiB (CPU-mapped), **KV 3,001.25
-MiB (5 full layers) + 306.25 MiB (25 SWA rings)**, compute buffers 926.39 +
-150.63 MiB, output buffer 1.00 MiB.
-
-**Measured decode: 139.9 tok/s** (`/completion` timings, temp 0, 256 tokens,
-deployed shape) — the ~4 B-active MoE is at home on the 3090. The model
-thinks: a 256-token budget on a short prompt can be consumed entirely by
-`reasoning_content` with empty `content` and `finish_reason: length`, so
-harness requests must budget for thinking (same law as the uncensored Swift
-payload). Vision probe on this build: solid-red 512 px PNG via the OpenAI
-`image_url` part → `Red`, `prompt_tokens` 151 (the image reached the model).
-
-Decode is expected ubatch-invariant (Swift-era 3090 law; the 139.9 t/s above
-is at ubatch 1024 — one rung measured, not a ladder).
-
-## Architecture facts
-
-Gemma4 MoE hybrid: 30 layers, 128 experts (8 active), n_embd 2816, 262,144
-native window, n_vocab 262,144. Attention is hybrid per
-`gemma4.attention.sliding_window_pattern` (read from the file at load): **25
-sliding-window layers (window 1024, `n_head_kv = 8`) + 5 full-attention layers
-(indices 5, 11, 17, 23, 29, `n_head_kv = 2`)**, head_dim 256 for K and V in
-both classes. The KV pool is TWO caches: the full-context cache scales with
-the 5 layers at the full 200,704 window; the SWA cache is a fixed 1024-token
-ring per sliding layer.
-
-**The rate law (measured, exact):** bytes/token = `n_kv_heads × 256 × 6.125 B`
-per layer at f16 K + q8_0 V — `attention_k_eq_v` does **NOT** shrink the
-allocation: `Vcur = Kcur` is a compute-graph mapping, the cache reserves K and
-V slots and, with `k_eq_v`, reserves the pair **twice** (6.125 = 2 × (2 B f16 +
-1.0625 B q8_0)). Verified against the boot: 5 × 2 × 256 × 6.125 = 15,680
-B/token × 200,704 = **3,001.25 MiB** ✓ and 25 × 1024 × 8 × 256 × 6.125 =
-**306.25 MiB** ✓. At q8_0 K + q8_0 V the rate is 2.125 B/elem: the same shape
-costs 1,041 + 106 = **1,147 MiB**, so the f16-K raise is **+2,161 MiB**. The
-`--cache-type-v` flag is honoured, not inert.
-
-The llama.cpp master build at commit `5fc4f3c` supports the arch
-(`src/models/gemma4.cpp` and `LLM_ARCH_GEMMA4 = "gemma4"` — both present in
-`/home/dank/GitPackages/llama-master` as read on 2026-10-18, `src/llama-arch.cpp:59`). Note: `grep -c`
-arch names on the packed binary returns 0 for EVERY arch (including the
-`qwen3_5` it serves) — check the source tree, not the binary.
-
-## MTP law
-
-MTP for Gemma4 is a SEPARATE file (`mtp-gemma-4-26B-A4B-it.gguf`, arch
-`gemma4-assistant`), loaded via `--spec-draft-model` beside
-`--spec-type draft-mtp` — unlike Swift, whose MTP head ships inside the main
-GGUF. The serve script gained `SPEC_DRAFT_FILE` for this. 2026-10-07 (Frank):
-**MTP disabled** — `SPEC_TYPE=none` strips both spec flags (flags move as a
-set). The assistant payload stays on disk; setting `SPEC_TYPE=draft-mtp` +
-`SPEC_DRAFT_FILE` re-enables it at the cost of ~1.9 GiB (boots at 319 MiB
-free).
-
-## Verification
-
-- `curl http://192.168.0.143:7771/health` → `{"status":"ok"}`
-- `/v1/models` reports `n_ctx: 200704`, `ftype: Q4_K - Medium`
-- Vision probe: solid-color PNG ≥ 256 px via OpenAI `image_url` part,
-  expect the exact color word.
-
-## Rollback
-
-Revert to the Swift trio: `MODEL_FILE=/home/dank/models/Swift15-UncIQ3XXS/…`,
-`CTX_SIZE=229376`, `PARALLEL=2`, in-file `SPEC_TYPE=draft-mtp` — the Swift
-payload remains on disk. See
-[[frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688]].

serverAppConfigs-docs/LLM/frank-zealot-llama-mimo-q5km.md

@@ -1,182 +0,0 @@
----
-type: reference
-title: "frank-zealot on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (retired)"
-description: "The retired 2026-10-02/03 frank-zealot runbook — the Q5_K_M rung at two wide unquantized-KV slots on the RTX 3090, superseded 2026-10-03 by the Q6_K file. Kept as the rollback reference: it carries the measured boot-pair table that prices every pool shape on this card, the slot-window contract, and the misbehaves tree."
-tags:
-  - llm
-  - frank-zealot
-  - dankclaw
-  - llama-cpp
-  - rtx3090
-  - mimo
-  - gguf
-  - kv-cache
-timestamp: 2026-10-03
----
-
-# frank-zealot — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090, retired)
-
-> **RETIRED 2026-10-02 → 2026-10-03** — superseded by
-> [[frank-zealot-llama-mimo-q6k-2x200704]]: the ~3.5 GiB this shape left free
-> bought the **Q6_K quant rung** (same runtime, same 2 × 200,704 pool, same
-> f16 KV), which measures 2,849 MiB free and 94.44 tok/s — 10 % slower
-> decode is the measured price of the higher rung on this bandwidth-bound
-> card. This file remains the rollback reference and the owner of the
-> measured boot-pair table. :7771 today runs
-> the PrismML 27B era of [[frank-zealot-llama-prism-bonsai2-pq20]] (build 10754,
-> 2 × 200,704 q8_0 KV, MTP off) — retired 2026-10-17 → 10-18 as a rollback
-> target; the active row today is build **10777** on
-> `Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf` at 2 × 262,144 exact f16 KV, MTP off by
-> measurement (AGENTS.md Runbook Index). The Q5_K_M
-> payload is still on disk (`stat -c %s` 2026-10-18 → 6876124704).
-
-> Its own history: one quant step down from [[frank-zealot-llama-mimo-q6k]]
-> (2026-10-01), resized twice on 2026-10-02 (three contexts with vision, then
-> the 2–3 GiB-band pool), and on 2026-10-03 morning reshaped to **two
-> conversations of 200,704 tokens with the KV VALUES unquantized** —
-> *"2 streams with 200700 tokens context or more… if we have extra space on
-> the 3090, unquantize the kv."* That shape measured 20,530 used / 4,046
-> apparent free (the true `nvidia-smi` free was 3,571 — see *Shape*).
-
-## Identity
-
-|Field|Value|
-|---|---|
-|Endpoint|`192.168.0.143:7771`, served alias `frank-small` in this era (`git show e5893dc:scripts/frank-small.env` → `ALIAS=frank-small`; `frank-zealot` only from 2026-10-06/10-09)|
-|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server` — unchanged across the quant swap|
-|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f` + the repo's `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`, 918,166,048 B, sha256 `ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576` — both at `/home/dank/models/MiMo-Q5KM/`, fetched and byte- and sha256-verified by `scripts/download-frank-zealot.sh`|
-|Unit|root `frank-zealot.service` (repo `scripts/frank-zealot.service`, `User=dank`, relinked), env `scripts/frank-zealot.env`, serve `scripts/serve-frank-zealot.sh`|
-|Repo pins|`tests/frank_zealot_unit.bats` (stub-argv seam + the VRAM arithmetic), `deploy_llm_zealot` gates in `scripts/deploy-remote.sh`|
-
-## Shape and the KV contract
-
-`--ctx-size 401408 --parallel 2` — llama.cpp divides the pool across slots,
-so each conversation gets `n_ctx_slot = 200704` (boot log 2026-10-03:
-`n_slots = 2, n_ctx_slot = 200704, kv_unified = 'false'`). `CTX_SIZE` MUST
-stay divisible by `PARALLEL`; the bats seam enforces it.
-
-The KV contract is Frank's, as of 2026-10-03: **both K and V unquantized
-(f16)** — `--cache-type-k f16 --cache-type-v f16`. The asymmetric contract it
-replaced (f16 K + `q8_0` V) had existed to make three slots fit; with two
-slots the exact-value budget opens, and 4 KiB/token × 8 layers × 2 of extra
-V traffic is the measured ~2 % decode tax. The serve script still passes both
-flags explicitly — the asymmetric-era law that a quantized V requires
-`--flash-attn on` stays pinned by the seam.
-
-**Pool sizes on this stack come from boots, not formulas.** The compute
-buffers, CUDA graphs and vision tower all bill separately, so every shape
-change is priced from measured rows:
-
-|Shape|KV dtype (K+V)|Projector|`nvidia-smi` used|Non-KV (weights + buffers + projector)|
-|---|---|---|---|---|
-|2 × 262,144 (pool 524,288)|f16 + q8_0|none|20,190 MiB|**7,390 MiB**|
-|3 × 196,608 (pool 589,824)|f16 + q8_0|f16|22,972 MiB|**8,859 MiB**|
-|3 × 180,224 (pool 540,672)|f16 + q8_0|f16|21,684 MiB|~8,419 MiB|
-|**2 × 200,704 (pool 401,408)**|**f16 + f16**|f16|**20,530 MiB**|**~7,986 MiB**|
-
-The 32 KiB/token full-attention KV price (8 layers × 4 KV heads × 256 dim ×
-2 B × 2 halves) prices the deployed pool at 401,408 × 32 KiB = **12,544 MiB
-of KV**; the card is 24,576 MiB, so the shape leaves **4,046 MiB free**
-(measured steady-state — the KV pool is allocated whole at boot, so this is
-the ceiling). The headroom above the 2–3 GiB floor is deliberate slack, not
-an invitation: another 32,768 tokens of pool (1 GiB) would trade the last
-safety margin for window the harness compacts before reaching anyway.
-
-**Why 2 × 262,144 with f16 V does not fit either.** The native window pair is
-524,288 tokens = 16,384 MiB of KV plus ~7,986 non-KV = 24.4 GiB: under the
-floor by a hair. 200,704 is the largest 4,096-aligned slice that keeps the
-measured band.
-
-## Measured on this shape (2026-10-03, 2 × 200,704 + vision, f16 KV)
-
-|Check|Result|
-|---|---|
-|Boot to healthy|~5 s, `n_slots = 2`, `n_ctx_slot = 200704`, `loaded multimodal model` logged|
-|VRAM|**20,530 MiB used / 4,046 MiB free** of 24,576 (`nvidia-smi`, steady state)|
-|Decode, temp 0, no thinking|**104.84 tok/s** (128-token completion at ~500-token depth; the 3 × 180,224 q8_0-V shape measured 106.8 — ~2 % is the honest price of exact V)|
-|Prefill|1,650 t/s on the same shallow 506-token probe (rates are depth- and prompt-dependent; not comparable across shapes)|
-|Batch geometry|`--batch-size 2048 --ubatch-size 1024` — ubatch 2048 failed the scratch boot; 1024 boots clean|
-|Vision|projector loads at boot (`loaded multimodal model`); red-PNG probe passed on 2026-10-02. Caveat discovered on 2026-10-03: probes must use a >= 256 px image — a 10x10 px PNG undersupplies image tokens and the model hallucinates ("it's a video")|
-|Tools / thinking|unchanged by the reslice: native `tool_calls`, thinking in `reasoning_content`|
-
-## What the slot geometry costs
-
-The per-slot window is the whole price, in both directions:
-
-- **A prompt over 200,704 tokens is a hard 400**, so `~/.omp/agent/models.yml`
-  sets `contextWindow: 200704` for this endpoint — the harness compacts before
-  it reaches the ceiling instead of discovering it.
-- **frank-dragoon's tallest pre-compaction prompt now fits here.** Compaction
-  for frank-dragoon sessions runs on this endpoint (`compactionModel` in
-  `models.yml`, still true 2026-10-18 at `~/.omp/agent/models.yml:107`); frank-dragoon compacts at 70 % of 262,144 = 183,501 tokens,
-  which did **not** fit the old 180,224 slot and fits 200,704 with 17,163
-  tokens to spare (2026-10-18: frank-dragoon's `contextWindow` is 155,648 —
-  `~/.omp/agent/models.yml:131` — so its compaction input is smaller still).
-  The 2026-10-02 window trim had restored the fit for
-  frank-archon's smaller 140,492-token trigger; the 2026-10-03 reslice widens
-  the fit to frank-dragoon itself. Nothing routes compaction automatically —
-  the point is the ceiling no longer forbids it.
-- Two simultaneous generations are the slot ceiling; a third request queues
-  behind a released slot (the mechanism measured on the 3-slot shape).
-
-## Client contract
-
-- Thinking arrives in **`reasoning_content`** (`--reasoning-format deepseek`),
-  the field `models.yml` reads for this endpoint.
-- **Thinking is off unless the request asks**: the MiMo GGUF template defaults
-  `enable_thinking` false, so `models.yml` ships
-  `chat_template_kwargs: {enable_thinking: true, preserve_thinking: true}` in
-  `extraBody`. A raw client that omits it gets a direct answer — measured, and
-  not a fault. Trivial questions produce no thinking even with the flag on.
-- Tools: `--jinja` + server-side parsing, verified 2026-10-02.
-- **Vision is on**, through the projector file (`--mmproj`). Use the
-  OpenAI-standard `image_url` content part — that path answers correctly
-  ("Red." on a solid-red PNG). The legacy top-level `images` field is **not**
-  read by `/v1/chat/completions`: the same request sent that way answered
-  "Brown.", i.e. it generated from text alone. `/completion` takes `images`
-  but refuses a request without `messages` in this build, so chat is the only
-  vision path.
-- Setting `MMPROJ_FILE=` (explicitly empty) in `scripts/frank-zealot.env` drops
-  `--mmproj` and returns ~1.4 GiB of the card to KV — the documented text-only
-  opt-out. `MMPROJ_FILE` pointing at a file that does not exist is a boot
-  refusal, never a silent fallback to blindness.
-
-## No spec decode
-
-The GGUF has no MTP tensors (the same fact that made SGLang's NEXTN a 0.00
-ghost), and the cross-model DFlash draft
-(`/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf` — no longer on
-disk: the 2026-10-18 census of `/home/dank/models/` has no `DFlash-Qwen35/`) is an
-optimization candidate, not a default — acceptance has to be measured on this
-traffic first. The bats seam fails if a draft flag ships unannounced.
-
-## Rollback
-
-|Target|How|
-|---|---|
-|Previous shape: 3 × 180,224 + q8_0 V (2026-10-02)|`scripts/frank-zealot.env` `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 MiB used, 2,441 free) and `contextWindow: 180224` in `models.yml` — choose it when three simultaneous conversations matter more than per-slot window or exact V|
-|Q6_K, same runtime|`scripts/frank-zealot.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + `MMPROJ_FILE` in that dir — the deployed rung **2026-10-03 → 10-05** ([[frank-zealot-llama-mimo-q6k-2x200704]]): at 2 × 200,704 it fits with 2,849 MiB free, no pool cut needed (the older "+1.4 GiB, drop 49,152 tokens" estimate held only for the 3-slot shapes)|
-|Two 262,144 windows|`CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=q8_0` (measured 20,190 MiB used, 4,386 free **with q8_0 V** — with f16 V the same shape is 16,384 MiB of KV + ~7,986 non-KV and does not fit) and `contextWindow: 262144` in `models.yml` — choose it when a single conversation needs the native window|
-|Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_ZEALOT_MMPROJ` at any real projector file or drop the gate in the same commit|
-|SGLang + INT8|[[frank-zealot-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 and **that stage dir no longer exists** (`ls /home/dank/.AGENTTRASH/`, 2026-10-18), so re-fetch with `old/download-frank-zealot-sglang-int8.sh`|
-
-## When it misbehaves
-
-- Port 7771 dark: `journalctl -u frank-zealot -n 50`; the serve script exits 1
-  naming the missing file if the binary or GGUF moved.
-- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM node
-  is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
-- Requests queue behind each other: two concurrent long generations are the
-  slot ceiling by contract (2026-10-03 shape); a third waits for a slot to
-  free (the mechanism measured on the 3-slot shape: 1.7 s for three, 3.0 s for
-  the fourth).
-- `400 … exceeds the available context size (200704 tokens)`: the conversation
-  outgrew ONE slot. This is not pool exhaustion — the pool is 401,408 tokens
-  and the per-slot window is the binding limit. Shorten the prompt, compact,
-  or roll back to the 262,144 shape above.
-- Vision answers a color it never saw, or claims the image "is a video":
-  either the client sent `images` instead of an `image_url` content part, or
-  the probe image is too small to supply real image tokens (use >= 256 px;
-  see *Client contract*).
-- Slow decode after a CTX or quant change: check `nvidia-smi` — pushing KV out
-  of VRAM into unified memory halves the rate silently.

serverAppConfigs-docs/LLM/frank-zealot-llama-mimo-q6k-2x200704.md

@@ -1,163 +0,0 @@
----
-type: reference
-title: "frank-zealot on Dankclaw — llama.cpp, MiMo-V2.6 Q6_K, 2 x 200704, f16 KV (RETIRED)"
-description: "The RETIRED frank-zealot runbook (2026-10-03 → 10-05 evening, rollback target) — llama.cpp master (CUDA sm_86) serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two conversations of 200,704 tokens with fully unquantized f16 KV, vision through the bundled projector, and the boot-pair arithmetic that prices every pool change. Superseded by the Swift-1.5-27B IQ3_S-mtp swap."
-tags:
-  - llm
-  - frank-zealot
-  - dankclaw
-  - llama-cpp
-  - gguf
-  - rtx3090
-timestamp: 2026-10-03
----
-
-# frank-zealot — llama.cpp MiMo Q6_K, 2 × 200,704, f16 KV
-
-> **RETIRED 2026-10-05 (evening).** Superseded by
-> [[frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688]] — the Swift-1.5-27B
-> RCO-IQ3_XXS-mtp swap (uncensored, MTP drafted). :7771 today runs
-> [[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML fork build 10754,
-> `Ternary-Bonsai-2-27B-PQ2_0.gguf`, 2 × 200,704 q8_0 KV, MTP off). Two facts for the trail:
-> committed earlier the same day (77decb5..380aafa) was **never deployed** —
-> THIS MiMo shape was the live payload when Swift cut over; (2) the KV
-> contract Frank set here (f16 K + f16 V, "no further quantization on this
-> endpoint") was superseded the same evening by the Swift swap's explicit
-> q8_0/q8_0 order. This page stays as the rollback and the measured
-> boot-pair record. Payload on disk (`stat -c %s`, 2026-10-18: 7793710624):
-> `/home/dank/models/MiMo-Q6K/`.
-
-## Identity
-
-| | |
-|---|---|
-| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
-| Endpoint | `http://192.168.0.143:7771` — alias `frank-small` on the wire in this era (`git show e5893dc:scripts/frank-small.env` → `ALIAS=frank-small`; the id became `frank-secondary` 2026-10-06 and `frank-zealot` 2026-10-09), `/v1` OpenAI surface |
-| Runtime | llama.cpp master, CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (f16 KV needs no flash-attn option; the disk build's `CMakeCache.txt` reads `GGML_CUDA_FA_ALL_QUANTS:BOOL=OFF` and that tree has no CUDA-side guard for the option — `ggml/src/ggml-cuda/fattn.cu:438-449`) |
-| Unit | root systemd `frank-zealot.service` ([`scripts/frank-zealot.service`](../../scripts/frank-zealot.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
-| Serve script | [`scripts/serve-frank-zealot.sh`](../../scripts/serve-frank-zealot.sh), tunables [`scripts/frank-zealot.env`](../../scripts/frank-zealot.env) |
-| Weights | `/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` — bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a…` (qwen3_5 hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, native 262,144, no MTP tensors) |
-| Projector | `…/MiMo-Q6K/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` — 918,166,048 B, sha256 `ff348f…` (hardlink of the Q5KM dir's copy, link-count-2 verified; the dir is self-contained) |
-| Fetch | [`scripts/download-frank-zealot.sh`](../../scripts/download-frank-zealot.sh) — `-f`, byte count + LFS sha256 verified, `--verify-only`; both files verified against the pinned sha on the host before this deploy |
-| Seam | `tests/frank_zealot_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, batch pair, projector opt-out) |
-
-## Shape and KV contract
-
-`--ctx-size 401408 --parallel 2` — llama.cpp divides the pool across slots,
-so each conversation gets `n_ctx_slot = 200704`. `CTX_SIZE` MUST stay
-divisible by `PARALLEL`; the seam enforces it. Both K and V are unquantized
-(f16) — the flags stay explicit because `--flash-attn` must not be lost if V
-ever goes back to `q8_0` (the serve script pins it, the seam pins the pin).
-
-**The Q6_K rung is a weights-side spend only.** File delta +917,585,920 B
-(+875 MiB) against Q5_K_M, same KV, same slot geometry, same batch pair.
-Priced from the measured Q5 boot at this shape (non-KV 7,986 MiB): the boot
-landed **21,276 MiB used / 2,849 MiB true free** — non-KV measured 8,732 MiB,
-inside the 2–3 GiB floor, so the pool stayed at 401,408 and the
-12,288-token step-down ladder was never climbed. Quote `nvidia-smi`'s `free`
-field, never `total − used`: the driver holds ~450 MiB of reserve the
-subtraction hides (that error once overstated this card's slack by a
-half-GiB).
-
-**Quant ceiling, evaluated and closed 2026-10-03 (Frank: stay here).**
-`Q8_0` is the only rung above this one on the Hub (9,545,979,424 B = 9,104
-MiB, +1,671 over Q6_K): at the live pool it would leave 1,178 MiB free —
-under the floor — and the calibrated bandwidth law (it predicted this file's
-94.4 t/s from Q5_K_M's 104.84 within 1 %) puts decode at ~72 t/s, a further
-−24 % for the near-lossless rung. Fitting it was priced two ways (trim the
-pool to 2 × 182,272 — which also breaks frank-dragoon's 183,501-token
-compaction-prompt fit by 1,236 tokens — or revert V to `q8_0`), and both
-were declined. Re-open only on evidence the Q6_K residual matters.
-
-**Pool prices on this card come from boots, never formulas** — the measured
-boot-pair table lives in [[frank-zealot-llama-mimo-q5km]] (*Shape and KV
-contract*) and still prices every rollback shape.
-
-## Compute-buffer geometry (2026-10-05)
-
-`BATCH_SIZE=4096 UBATCH_SIZE=2048` — both doubled from 2048/1024, and the trade
-measured rather than assumed, because the predecessor shape recorded **"ubatch
-2048 failed the scratch boot"** and this card has less free space than it did.
-
-**That failure was a different shape, and it did not reproduce here.** The
-service itself is the bench (one card holds one server, so no scratch port):
-the old rung was re-booted through a transient `FRANK_ZEALOT_ENV_FILE` drop-in
-and reproduced the 2026-10-03 boot **byte-identically** — 21,276 MiB used /
-2,849 MiB free — which makes the ubatch the only variable in the comparison.
-
-|Probe|ubatch 1024|ubatch 2048|Δ|
-|---|---|---|---|
-|Prefill, 8,928 tokens|3,972 / 4,077 t/s|**4,207 / 4,212 t/s**|**+4.6 %**|
-|Decode, 128 tokens at that depth|91.87 / 93.01 t/s|93.81 / 94.27 t/s|noise|
-|Prefill, 279 tokens|2,675.7 t/s|2,658–2,811 t/s|noise|
-|VRAM used / free|21,276 / 2,849 MiB|**21,846 / 2,279 MiB**|**−616 MiB**|
-
-Three readings:
-
-- **The gain only exists above the old ubatch.** A 279-token prompt is shorter
-  than either micro-batch, so it cannot see the knob — and it did not move.
-  That is the control proving the 8.9k delta is geometry, not drift.
-- **Decode is flat**, as every prior sweep on this card found. Ubatch is a
-  prefill knob.
-- **The price is 616 MiB out of a 2.8 GiB margin.** 2,279 MiB free is still
-  inside the 2–3 GiB band, at its bottom edge. `4096/1536` is the middle rung,
-  and the pool's 12,288-token (384 MiB) ladder in
-  [[frank-zealot-llama-mimo-q5km]] is the other way to buy room back.
-
-**Why 2048 is right HERE while 512 was right on the R9700 sibling: the curve is
-backend-specific.** The 2× R9700 layer split peaks at ubatch 512 and is 10 %
-off that peak by 2048 ([[frank-medium-batch-ubatch]]); on this CUDA card the
-curve climbs monotonically to 2048 (710 → 1,188 t/s on the 35B MoE sweep,
-1,944 → 2,423 t/s on the Ornith 8k sweep, both with decode flat). Never carry a
-ubatch verdict across backends.
-
-The logical `--batch-size` is VRAM- and throughput-neutral — two sweeps and this
-A/B agree — so 4096 costs nothing and buys nothing by itself; it is here because
-that is the pair the curve was measured at.
-
-## Measured on this shape (2026-10-03, Q6_K, 2 × 200,704, f16 KV)
-
-|Check|Result|
-|---|---|
-|Boot to healthy|model + projector resident in ~2.3 s (load log `0.02.27`), `n_slots = 2`, `n_ctx_slot = 200704`, `loaded multimodal model`, health 200; deploy e2e gate 33/33|
-|VRAM|**21,276 MiB used / 2,849 MiB free** of 24,576 (`nvidia-smi` `free`, steady state) — inside the 2–3 GiB floor with exact V AND the higher rung. Superseded 2026-10-05 by the batch doubling: **21,846 / 2,279** — see *Compute-buffer geometry*|
-|Decode, temp 0, no thinking|**94.44 tok/s** on the identical 506-token probe that measured 104.84 on Q5_K_M — **−10 %**. Decode on this card is weight-bandwidth-bound, and Q6_K streams ~10 % more bytes per token; the quant rung buys quality with decode rate, measurably|
-|Prefill|2,229.9 t/s (same shallow probe; rates are depth-dependent, not comparable across probes)|
-|Vision|512×512 solid-red PNG → **`Red`** (277 prompt tokens — the image really reached the tower). See the probe-size trap in *Client contract*|
-|Tools / thinking|native `tool_calls`, thinking in `reasoning_content`, thinking OFF unless `enable_thinking: true` (MiMo template default; `models.yml` ships it in `extraBody`)|
-
-## Client contract
-
-llama.cpp surface, live-verified on this model family: model id `frank-zealot`,
-native tools via `--jinja` (`tool_calls` in the response), thinking in
-**`reasoning_content`**, vision via `image_url` content parts. Two vision
-traps, both measured here:
-
-1. A top-level `images` field on `/v1/chat/completions` is silently ignored
-   and the model invents a color.
-2. **The probe image must be big enough to supply real image tokens.** A
-   10×10 px solid-red PNG came back "The image is a video, so I can't see its
-   content" — the engine itself warns at boot that Qwen-VL wants ≥ 1024 image
-   tokens, and a postage-stamp image lands one patch below usable signal.
-   512×512 answers `Red` in 19 tokens. Always probe vision with a ≥ 256 px
-   image; a wrong-color answer from a tiny image is the probe's fault, not
-   the endpoint's.
-
-`~/.omp/agent/models.yml` sets `contextWindow: 200704` — the PER-SLOT window,
-so the harness compacts before llama.cpp answers a longer prompt with a 400.
-
-## Rollback
-
-|Target|How|
-|---|---|
-|Q5_K_M at the same shape|`scripts/frank-zealot.env` `MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf` (+ `MMPROJ_FILE` in that dir) and the deployer default — buys back 875 MiB (3,571 free measured) **and the 10 % decode rate** (104.84 tok/s measured). Payload sha-verified on disk|
-|3 × 180,224 + q8_0 V|`CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 used / 2,441 free **with Q5_K_M** — Q6_K does NOT fit that shape inside the floor; pair them explicitly)|
-|SGLang + INT8|[[frank-zealot-sglang-mimo-v26]] — the 2026-10-02 staging dir `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` is **gone** (`ls /home/dank/.AGENTTRASH/` on 2026-10-18 lists only the 20261005/06/07/10/17 batches), so this arm needs `old/download-frank-zealot-sglang-int8.sh`; the `sglang-cu128` venv is still on disk|
-
-## When it misbehaves
-
-Same decision tree as [[frank-zealot-llama-mimo-q5km]] (*When it misbehaves*):
-`No CUDA GPUs are available` ⇒ `nvidia-modprobe -u -c=0`; a 400 naming
-200,704 ⇒ the conversation outgrew one slot, not the pool; slow decode after
-a shape change ⇒ check `nvidia-smi` for KV pushed into unified memory; queue
-behind two long generations ⇒ the slot ceiling is two, by contract.

serverAppConfigs-docs/LLM/frank-zealot-llama-mimo-q6k.md

@@ -1,120 +0,0 @@
----
-type: reference
-title: "frank-zealot on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF (retired)"
-description: "The retired 2026-10-01 frank-zealot runbook — the first llama.cpp + GGUF shape on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V) that motivated the move off SGLang, the VRAM fit at Q6_K, and what to check when it misbehaves. Superseded the same night by the Q5_K_M payload."
-tags:
-  - llm
-  - frank-zealot
-  - dankclaw
-  - llama-cpp
-  - rtx3090
-  - mimo
-  - gguf
-  - kv-cache
-timestamp: 2026-10-01
----
-
-# frank-zealot — llama.cpp MiMo-V2.6 Q6_K (RTX 3090)
-
-> **RETIRED 2026-10-01 → 10-02** (one night): superseded by
-> [[frank-zealot-llama-mimo-q5km]], the same runtime and shape one quant rung
-> down. The Q6_K file stays at `/home/dank/models/MiMo-Q6K/` as the rollback
-> target (on disk 2026-10-18, `stat -c %s` → 7793710624). :7771 today runs
-> [[frank-zealot-llama-prism-bonsai9-mtp]] (llama.cpp PrismML fork build
-> **10777**, `Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, 2 × 262,144,
-> exact f16 K + f16 V, MTP off by measurement).
->
-> Original order (Frank, 2026-10-01 night): move this endpoint to the GGUF
-> path for **asymmetric KV** — unquantized K + q8_0 V — which only llama.cpp
-> exposes per-tensor. The per-tensor lever stayed; this pairing did not — the
-> live endpoint bills **exact f16 K + f16 V** — 32,768 B/token = 16,384 MiB
-> over the 524,288-position pool of two 262,144 windows
-> (`scripts/frank-zealot.env:57-58,80-81`).
-
-## Identity
-
-|Field|Value|
-|---|---|
-|Endpoint|`192.168.0.143:7771`, served alias `frank-small` in this era (`ALIAS` reached `frank-secondary` 2026-10-06 and `frank-zealot` 2026-10-09, commit `7c77326`)|
-|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`|
-|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a2ddf2cbb450d1af1ac3860ec769d3575bafa70692ee5609bda3fad7d`, at `/home/dank/models/MiMo-Q6K/`|
-|Unit|root `frank-zealot.service` (repo `scripts/frank-zealot.service`, `User=dank`, relinked), env `scripts/frank-zealot.env`, serve `scripts/serve-frank-zealot.sh`|
-|Repo pins|`tests/frank_zealot_unit.bats` (stub-argv seam), `deploy_llm_zealot` gates in `scripts/deploy-remote.sh`|
-
-## Shape and the KV contract
-
-`--ctx-size 524288 --parallel 2` — llama.cpp divides the pool across slots,
-so each session gets `n_ctx_slot = 262144`, the model's full native window
-(boot log confirms `n_slots = 2, n_ctx_slot = 262144, kv_unified = 'false'`).
-
-Frank's KV contract: **K unquantized (f16), V `q8_0`** —
-`--cache-type-k f16 --cache-type-v q8_0`. Two hard dependencies ride on it:
-
-- **`--flash-attn on` is mandatory**: quantized V exists only on the FA
-  path. Without the flag the engine falls back silently and the contract
-  becomes a lie. The serve script pins the trio; the bats seam pins the
-  script.
-- The VRAM budget: attention KV lives in the 8 full-attention layers only
-  (4 KV heads x 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token
-  (34 B per 32-element block) ⇒ ~24.5 KiB/token over the 524,288-token pool
-  ≈ **12.6 GiB**. Weights 7.8 + KV 12.6 + compute buffers/graphs ≈
-  **20.9 GiB measured** against 24.5 GiB — the "room for checkpoints" the
-  order asked for. If graph capture OOMs, halve `CTX_SIZE`; never quantize
-  K to fix it.
-
-## Client contract
-
-- Thinking arrives in **`reasoning_content`** (`--reasoning-format
-  deepseek`) — the same field SGLang's qwen3 parser used, so `models.yml`
-  `reasoningContentField` did NOT move. Live-verified 2026-10-01 22:57.
-- **Thinking is off unless the request asks**: the MiMo GGUF template
-  defaults `enable_thinking` false; `models.yml` now ships
-  `chat_template_kwargs: {enable_thinking: true, preserve_thinking: true}`
-  in `extraBody`. A raw client that omits it gets a no-thinking answer
-  (measured — this is not a fault).
-- Tools: `--jinja` + server-side tool parsing, verified
-  `tool_calls` round-trip 2026-10-01 23:0x.
-- **Text-only in this era.** The GGUF carries no vision projector; the
-  in-checkpoint tower existed only in the SGLang safetensors, so `models.yml`
-  carried `input: [text]`. Vision returned with the next payloads: the endpoint
-  is `input: [text, image]` today (`~/.omp/agent/models.yml:214`).
-
-## No spec decode (yet)
-
-The GGUF has no MTP tensors (same fact that killed SGLang NEXTN), and the
-cross-model DFlash draft (`/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf`,
-Q8_0 — that path is no longer on disk; the 2026-10-18 census of
-`/home/dank/models/` has no `DFlash-Qwen35/`) is an optimization-loop candidate, not a default — `--spec-type
-draft-dflash` needs acceptance measured on this traffic first. The bats
-seam fails if a draft flag ships unannounced.
-
-## Build
-
-No ccache, CUDA 12.4 system nvcc with `--gcc-install-dir`-free gcc-13 host
-compiler (nvcc 12.4 rejects gcc ≥ 14):
-
-```
-cmake -B build-cuda -S . -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON \
-  -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13 \
-  -DLLAMA_CURL=OFF -DGGML_CCACHE=OFF
-cmake --build build-cuda -j 16 --target llama-server
-```
-
-The SGLang venv (`/home/dank/GitPackages/sglang-cu128`) is still on disk; the
-INT8 AutoRound payload (`/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`)
-is not — it was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/`
-on 2026-10-02 and that stage dir is gone as of 2026-10-18, so this rollback
-re-fetches with `old/download-frank-zealot-sglang-int8.sh`: repoint the unit's serve script and env from
-git history (`git show frank-dragoon-gufo-q4kxl-20261001` era is BEFORE this;
-use the parent of the 2026-10-01 night swap commit).
-
-## When it misbehaves
-
-- Port 7771 dark: `journalctl -u frank-zealot -n 50`; the serve script
-  exits 1 naming the missing file if the binary or GGUF moved.
-- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM
-  node is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
-- Requests hang at first token while `/health` is 200: both slots busy —
-  two concurrent long generations are the ceiling by contract.
-- Slow decode after a CTX change: check `nvidia-smi` — swapping KV out of
-  VRAM into unified memory halves the rate silently.

serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai2-pq20.md

@@ -1,226 +0,0 @@
----
-type: reference
-title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10754), Bonsai-2-27B PQ2_0, 2 x 200704, q8_0 KV (RETIRED, rollback target)"
-description: "The frank-zealot runbook of 2026-10-17 → 2026-10-18 (RETIRED — rollback target) — the PrismML fork build 10754 of llama.cpp serving prism-ml/Ternary-Bonsai-2-27B-gguf at PQ2_0 on the RTX 3090: two conversations of 200,704 tokens with q8_0 K + q8_0 V (f16 KV does not fit this card at this window), vision through the publisher's Q8_0 Qwen3-VL projector, MTP impossible (no tensors in the file), and the measured four-arm VRAM census that priced the shape. Superseded by the 9B MTP export at 3 × 262,144."
-tags:
-  - llm
-  - frank-zealot
-  - dankclaw
-  - llama-cpp
-  - gguf
-  - rtx3090
-timestamp: 2026-10-17
----
-
-# frank-zealot — llama.cpp (PrismML) Bonsai-2-27B PQ2_0, 2 × 200,704, q8_0 KV
-
-**RETIRED 2026-10-17 → 2026-10-18** (AGENTS.md Runbook Index), superseded by
-[[frank-zealot-llama-prism-bonsai9-mtp]] — prism-ml/Ternary-Bonsai-2-**9B**
-PQ2_0 **with the MTP head**, THREE conversations at the native 262,144 window
-on fork build 10777. This era stays the **rollback target**: the payload
-(`/home/dank/models/Bonsai2-PQ2_0/`) and the 10754 fork tree
-(`~/GitPackages/llama-prism/`) are both still on disk. It owns the laws below —
-the 34,816 B/token KV figure over 16 full-attention layers, the 25,088 MiB f16
-OOM wall, and the two-boot VRAM pricing that every later shape on this card
-reused.
-
-## 2026-10-18 bench window — production unit stopped, candidate arm on 7771
-
-As of 2026-10-18 frank-zealot sits in a deliberate, dated **bench window**
-(fleet Benchmarking Policy: stop the production unit, bench candidate boots
-against the service itself). The journal shows the stop at 03:38:31 and the
-known llama.cpp shutdown shape — SIGTERM ignored past `TimeoutStopSec`,
-`State 'stop-sigterm' timed out. Killing.` at 03:40:01 — after which the unit
-reports `failed` (`is-enabled` → enabled); self-healing is merely parked
-while the window runs. What answers 7771 is a **hand-booted candidate arm**
-(`llama-prism-b10777/llama-server` on the 9B MTP file
-`Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, alias
-`frank-zealot`), not this runbook's payload: candidate-arm flags are not a
-state — the pid and shape moved between two censuses ten minutes apart (one
-argv read `--ctx-size 786432 --parallel 3 --spec-type draft-mtp`; a later
-`/props` read `n_ctx 262144`, `speculative.types=none`); quote neither, and
-expect `/v1/models` to report the candidate until the window closes. The
-window's remedy is the redeploy: `./scripts/deploy.sh dankclaw --service
-frank-zealot`. The production trio below is unchanged — the last PRODUCTION
-boot under the unit (03:14–03:38 the same morning) printed the contract
-exactly: `n_ctx = 401408`, `n_ctx_seq = 200704`, `CUDA0 KV buffer size =
-13328.00 MiB`, 2 slots (journal).
-
-**Window CLOSED 2026-10-18:** that candidate became production. The redeploy
-shipped the 9B MTP export at `3 × 262,144` on fork build 10777 — see
-[[frank-zealot-llama-prism-bonsai9-mtp]]; this payload is the rollback target.
-
-## Identity
-
-| | |
-|---|---|
-| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB (24,123 MiB visible to the engine) |
-| Endpoint | `http://192.168.0.143:7771` — served model id **`frank-zealot`** (`ALIAS`), `/v1` OpenAI surface |
-| Runtime | **PrismML fork build** `/home/dank/GitPackages/llama-prism/llama-server` — vendor release, `--version` = `0.2.0-dev (build 10754, commit 2459f68b5)`, sha256 `12ae5baba033d8cd…`. NOT a preference: see *Why the fork* |
-| Unit | root systemd `frank-zealot.service` ([`scripts/frank-zealot.service`](../../scripts/frank-zealot.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
-| Serve script | [`scripts/serve-frank-zealot.sh`](../../scripts/serve-frank-zealot.sh), tunables [`scripts/frank-zealot.env`](../../scripts/frank-zealot.env) |
-| Weights | `/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-PQ2_0.gguf` — 7,206,168,928 B, sha256 `3907dc1658db1f78a9826bf8d5bcb8dc65db0d466388937af57f2294fae62ec1` |
-| Projector | `Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf` — 629,246,976 B (full sha256 pinned in the fetch script), Qwen3-VL tower; **873.10 MiB worst-case VRAM, loaded AFTER the engine's fit check** |
-| Fetch | [`scripts/download-frank-zealot-bonsai2.sh`](../../scripts/download-frank-zealot-bonsai2.sh) — `-f`, byte-count + sha256 verified, idempotent, `--verify-only`, no token on the command line |
-| Seam | `tests/frank_zealot_unit.bats` (argv assertions against a stub engine) |
-
-## Why the fork — the format is the lock
-
-Read from the file's own metadata, not the model card:
-
-|Key|Value|
-|---|---|
-|`general.architecture`|`qwen35` (hybrid: Gated DeltaNet + softmax attention)|
-|`qwen35.block_count`|64, `full_attention_interval` 4 ⇒ **16 full-attention + 48 DeltaNet layers**|
-|Attention|24 heads / **4 KV heads** × key 256 / value 256; `rope.dimension_sections` [11,11,10,0], `freq_base` 1e7, partial rope `dimension_count` 64|
-|Widths|embedding 5,120, feed-forward 17,408|
-|`qwen35.ssm.*`|conv_kernel 4, state_size 128, group_count 16, **time_step_rank 48**, **inner_size 6,144**|
-|Window / vocab|`context_length` 262,144; vocab 248,320 (gpt2 BPE, pre `qwen35`), eos 248046|
-|`general.file_type`|**141 = PQ2_0 — 2.13 bpw (group 128)**|
-|Tensor histogram|851 tensors: **`pq2_0` ×402** (custom GGML type **142**), f32 ×353, bf16 ×96|
-|`prism.hadamard.*`|transform `normalized-sylvester-walsh-hadamard`, block_size 1024, axis `input-last-dimension`, sign_mode explicit, **401 named weights**, sign widths [5120, 6144, 17408], 28,672 sign values, inverse applied on `token_embd.weight`, `gdn_v_grouped` true|
-|Sampler metadata|`general.sampling.temp` 1.0 / `top_p` 0.95 / `top_k` 20|
-|Chat template|8,952 chars; reads `enable_thinking`, **`reasoning_effort` ∈ {`xhigh` (default), `medium`, `low` — anything else is `raise_exception`**, `preserve_thinking`, `tools`/`tool_call`, image + video content parts|
-
-Stock llama.cpp cannot load this file: type 142 is not in its enum (`gguf-py`'s
-stock `GGMLQuantizationType` raises `ValueError: 142 is not a valid…` on the
-same read), and even a loader that parsed it would ignore the
-`prism.hadamard.*` dequant transform the weights depend on. Same class of lock
-as the ROCmFPX custom-type era. The vendor build 10754 already carries the
-September upstream fixes (AVX-512 load segfault, and a `GGML_CUDA_FA_ALL_QUANTS`
--capable CUDA backend — proven by q8_0 V serving under flash attention at the
-correct 34,816 B/token, below).
-
-## VRAM census (2026-10-17, four arms, production stopped)
-
-The engine's own `common_memory_breakdown_print` ledger + settled `nvidia-smi`,
-all arms `--n-gpu-layers 99 --flash-attn on --batch-size 4096 --ubatch-size 2048`,
-mmproj attached:
-
-|Shape|KV|projected `model + context + compute`|settled used / free|verdict|
-|---|---|---|---|---|
-|**2 × 200,704 (pool 401,408)**|**q8_0 / q8_0**|21,439 = 6,539 + 13,627 + 1,272|**22,664 / 1,461**|**DEPLOYED**|
-|2 × 200,704|f16 / f16|33,198 = 6,539 + 25,387 + 1,272|—|`cudaMalloc failed: out of memory` on the **25,088.00 MiB** KV request|
-|1 × 262,144|q8_0 / q8_0|16,905 = 6,539 + 8,853 + 1,512|18,130 / 5,995|fits, but one slot serializes compaction|
-|2 × 262,144|q8_0 / q8_0|≈25,758 (arithmetic, same terms)|—|does not fit (f16 analogue measured 41,118)|
-|2 × 200,704|q4_0 / q4_0|15,167 = 6,539 + 7,355 + 1,272|16,392 / 7,733|the roomy fallback rung|
-
-**KV rate law (confirmed to the MiB):** 16 full layers × 4 KV heads × 256 dim =
-16,384 elements/token/cache; q8_0 bills 17,408 B per cache ⇒ **34,816 B/token**,
-so the 401,408 pool = **13,328.00 MiB** — the ledger printed exactly that
-(`CUDA0 KV buffer size = 13328.00 MiB`). f16 doubles it to 25,088 MiB, which is
-the wall. The 48 DeltaNet layers bill a fixed recurrent state per **slot**:
-context 13,627 − KV 13,328 = **299 MiB for two slots (~150 MiB each)** — pool
-arithmetic alone under-prices the shape by that much.
-
-**The projector is outside the fit check.** `[mtmd] estimated worst-case memory
-usage of mmproj is 873.10 MiB` is printed before fitting but not billed in it:
-settled used (22,664) − projection (21,439) ≈ 1.2 GiB is that projector plus the
-CUDA context. This is the difference between "2.6 GiB free on paper" and the
-real **1.4 GiB** — the same late-allocation failure class that killed the
-DSpark drafter boot in the llama.cpp eras. Any resize must re-read the settled
-`nvidia-smi`, never the projection.
-
-## Measured rates (greedy, `Server-Timing`-free `/completion` `timings`)
-
-|Probe|Result|
-|---|---|
-|Short, warm, single stream (128 tok)|**76.6 t/s** decode (the 9B Q6_K predecessor ran 94.4 — the ternary 27B is ~19 % slower, weight bandwidth + Hadamard dequant)|
-|First request after boot|2.6 t/s prefill / 37.7 t/s decode — **warmup artifact; never report the first request** (measured twice on this payload)|
-|Prefill at ~35k depth|**1,258.4 t/s** (27.8 s)|
-|Prefill at 166,027-token depth|**691.7 t/s** (240.0 s), zero errors, slot intact|
-|Two concurrent streams, 256 tok each|**54.7 + 54.7 = 109.4 t/s aggregate** — `--parallel 2` really batches|
-|Long-context retrieval (25,960-token prompt, needle at position 180 of 300 passages)|exact: *"Frank keeps the fleet in git, only through it."*|
-
-Production boot (the 19:44 cutover deploy, same binary/flags/file): **75.8 t/s**
-single-stream decode at a short prompt — within 1 % of the trial arm, so the
-ladder above carries to the shipped shape.
-
-## Behavior contract (probes, not assumptions)
-
-- **Reasoning** lands in `reasoning_content` (`--reasoning-format deepseek`) ✓.
-- **Tool calls**: a `get_weather` request returns `finish_reason: tool_calls`
-  with `arguments: {"city":"Seattle"}` — valid JSON, no loop, no HTTP 500. The
-  fork's `KNOWN_ISSUES.md` warning about malformed/looping tool calls did not
-  reproduce on build 10754 at this shape.
-- **Vision**: a 512 px solid-red PNG answers **"Red"** — but ONLY with a
-  thinking budget ≥ ~512 tokens. At `max_tokens: 64` the request returns
-  `finish_reason: "length"` with EMPTY `content` and half a thought in
-  `reasoning_content`: thinking tokens count against the budget, so a
-  short-budget vision probe reads as "the model is blind". Images do reach the
-  model (a 512 px image bills ~280 prompt tokens: 322 total vs 42 for the text
-  alone). Keep the ≥256 px probe rule from the earlier eras.
-- **Log noise law**: `/completion` (text context) at `VERBOSITY=4` prints
-  `failed to parse reasoning content (response_format json_schema is currently
-  only supported with parse_reasoning=true), content will be stored in
-  'content' field` (`probe_request.cpp:789`) whenever the model closes a raw
-  prompt with ``. It is **cosmetic** — the full chain of thought
-  is in `content`. The OpenAI `/v1/chat/completions` path parses it into
-  `reasoning_content` correctly, which is the path the harness and every
-  consumer uses. Do not diagnose this as a fault, and do not drop verbosity
-  below 4 — the KV/buffer ledger in the journal is what VRAM forensics read.
-- **Long raw-prompt completions can come back empty**: at ~26k+ tokens a bare
-  `/completion` request generated 0–1 tokens with no error (the model emits an
-  end-of-turn marker immediately, and a text context has no role frame to keep
-  it talking). Chat traffic is unaffected; nothing in the harness posts raw
-  `/completion` at depth.
-
-## Ops notes
-
-- **Trial-window law (cost one boot window on 2026-10-17):** a scratch env file
-  must use the serve script's OWN variable names — `CACHE_TYPE_K`/`CACHE_TYPE_V`,
-  `BIND_ADDR`, `NGL`. An invented `KV_CACHE_TYPE` is silently swallowed by the
-  script's `: "${CACHE_TYPE_K:=f16}"` default and the arm boots **f16**, which
-  for this payload is an OOM that looks like a model problem.
-- **`/metrics` needs `--metrics` (fixed 2026-10-18):** the fork answers **501 Not
-  Implemented** without the flag, and Dankclaw's Prometheus scrapes `/metrics`, so
-  `up{job="frank-zealot"}` has been 0 for the whole retained metric history — the
-  oldest sample (2026-10-11) is already 0 — while the endpoint itself answered
-  `/health` 200 the entire time. `METRICS=1` in `scripts/frank-zealot.env` now
-  ships the flag and the seam asserts it reaches the argv. halogen serves
-  `/metrics` natively, which is why that target was always green.
-- **Base alignment:** Bonsai-2 is the BASE (censored) Qwen3.8-27B lineage, so a
-  refusal is possible on a compaction or subagent request. The CRACK re-export
-  (`dealignai/Bonsai-2-27B-Ternary-CRACK-GGUF`, already on disk at
-  `/home/dank/models/Bonsai2-CRACK/Bonsai-2-27B-PQ2_0-CRACK.gguf`, sha256
-  `5b24ea3e…`) is a **one-line `MODEL_FILE` swap**: the header census proves
-  byte-identical geometry — 851 tensors, the same `{142:402, f32:353, bf16:96}`
-  histogram, 64 blocks, vocab 248,320, the same 8,952-char template. The VRAM
-  ledger does not move. If a refusal ever surfaces, that is the fix; there is
-  also an MTP-bearing CRACK re-export upstream
-  (`ManniX-ITA/…-CRACK-MTP-GGUF`, 7,657,489,920 B, +451 MB head) if decode ever
-  needs the drafter — untested here.
-- **Compaction hub unchanged:** all three endpoints still route
-  `compactionModel` here; 200,704 per slot is the fleet's only window that fits
-  a worst-case compaction input, and `--parallel 2` keeps compaction off the
-  critical path of a live generation.
-- **Harness side** (`~/.omp/agent/models.yml`): `contextWindow: 200704` and
-  `maxTokens: 16384` unchanged; `extraBody` now quotes the payload's own
-  1.0 / 0.95 / 20 with `preserve_thinking: true`. The `reasoningEffortMap` fold
-  became **load-bearing** (the template raises on any level outside
-  `xhigh|medium|low`) — an unmapped level is a hard 500 on the role's first
-  request, naming Jinja rather than the model.
-- **Rollback:** revert the swap commit. The Qwen3.5-9B Q6_K payload, its mmproj
-  and the master CUDA build all remain on disk
-  (`/home/dank/models/Qwen35-9B-HauhauCS-Q6K/`,
-  `~/GitPackages/llama-master/build-cuda/`), and the retired era keeps its own
-  page: [[frank-zealot-llama-qwen35-9b-q6k-2x200704]].
-- **Deployed 2026-10-17 19:44** (`deploy.sh dankclaw --service frank-zealot
-  --service homepage`): boot ledger 22,664 used / 1,461 free, engine fit
-  projected 21,439 and left 2,119 ≥ 1,872 MiB, KV 13,328.00 MiB at
-  `n_ctx_seq = 200,704`, `n_slots = 2`, projector loaded; live probes green
-  (`/v1/models` serves `frank-zealot`, exact `pong` + `reasoning_content`,
-  `tool_calls` `{"city":"Seattle"}`, 512 px PNG → **Red**). e2e gate 33/33.
-- **Picker row is live state:** the OpenWebUI `model` row's
-  `meta.description` named the 9B, so it was re-keyed in the database too
-  (backup `webui.db.bak-20261017a`). The row was written WITHOUT restarting
-  `open-webui` — the cached list refreshes on its next restart, and forcing one
-  would have logged every family session out for a cosmetic string. See
-  [[OpenWebUI]] for the picker trio.
-
-## Related
-
-- [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — the replaced era (rollback target)
-- [[frank-dragoon-flash-server-v2-swift15]] — the abliterated sibling on the iGPU
-- [[frank-archon-radiance-qwen38-iq4r]] — the censored-base flagship this payload shares base alignment with
-- [[deployment]] — deploy flow and incidents

serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai9-mtp.md

@@ -1,362 +0,0 @@
----
-type: reference
-title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10777), Bonsai-2-9B PQ2_0 with the MTP head, 2 × 262144, exact f16 KV (ACTIVE)"
-description: "The ACTIVE frank-zealot runbook (since 2026-10-18) — the PrismML fork build 10777 of llama.cpp serving prism-ml/Ternary-Bonsai-2-9B-gguf at PQ2_0 on the RTX 3090: TWO conversations of 262,144 tokens (every slot at the model's trained window) over an UNQUANTIZED f16 K + V cache, vision through the publisher's Q8_0 Qwen3-VL projector, and the boot-arm census that put speculative decoding OFF by measurement even though the checkpoint carries the MTP head."
-tags:
-  - llm
-  - frank-zealot
-  - dankclaw
-  - llama-cpp
-  - gguf
-  - rtx3090
-  - mtp
-  - spec-decode
-timestamp: 2026-10-18
----
-
-# frank-zealot — llama.cpp (PrismML) Bonsai-2-9B PQ2_0 + MTP head, 2 × 262,144, exact f16 KV
-
-Live since 2026-10-18, replacing the
-[[frank-zealot-llama-prism-bonsai2-pq20]] era (Ternary-Bonsai-2-**27B** PQ2_0 at
-2 × 200,704). The original trade: the 9B bills **half** the KV per token of the
-27B, and that is what funded **three** conversations at the model's **native
-262,144** window where the 27B afforded only two shortened ones (Frank's ask:
-*"if you can possibly fit a third context, do it"*).
-
-The shape moved again the same evening (Frank: *"shift frank-zealot to 2
-parallels of unquantized kv"*). The KV cache went **exact** — f16 K + f16 V, no
-quantization error anywhere in the attention state — and the **slot count paid
-for it: 3 → 2**. The per-slot contract never moved; every conversation is still
-262,144, still the trained window. What the dtype bought is fidelity, and what
-it costs is one concurrent conversation plus the bytes attention reads per
-generated token.
-
-## Payload identity
-
-| Item | Value |
-|---|---|
-| Runtime | `/home/dank/GitPackages/llama-prism-b10777/llama-server` — PrismML **fork** build, `--version` → `0.2.0-dev (build 10777, commit e3de71522)` |
-| Checkpoint | `/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf` — 2,675,410,720 B, Hub-LFS sha256 `c79fcdfaf6b7bf47089b50dadcfd11f9ce7d3e96adf3bbefde7912e5ef69e22b` |
-| Vision | `/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf` — 624,229,664 B, Hub-LFS sha256 `e0cb6f6988cea323ef798cd77d019bbb819af75582f034eaab0e7232314953db` |
-| Fetch | `scripts/download-frank-zealot-bonsai9.sh [DEST]` (`--verify-only`; `-f`, byte-count and LFS sha256 per file) |
-| Alias | `frank-zealot` (`ALIAS`); the SERVICE/unit/scripts keep the StarCraft name, only the wire id is the model id |
-| Port | 7771, root systemd unit `frank-zealot.service` (relinked), `TimeoutStopSec=180` |
-
-### The runtime lock has two tumblers now
-
-1. **The tensor type.** `general.file_type 141` with **202 tensors of the fork's
-   PQ2_0 type** (of 442 total; the rest are f32 norms/scalars and 8 q8_0 in the
-   MTP block) plus `prism.hadamard.*` metadata
-   (`normalized-sylvester-walsh-hadamard`, `block_size 4096`,
-   `axis input-last-dimension`, `sign_mode explicit` over 201 weight names,
-   inverse on `token_embd.weight`, `gdn_v_grouped true`). Stock llama.cpp
-   refuses the custom type at load; a loader that ignored the Hadamard metadata
-   would dequant **wrong**, silently. Same lock class as the ROCmFPX era.
-2. **The build floor.** The fork's card gates **MTP beside a quantized V cache**
-   on `prism-b10776-d00e252` or newer, which is why the prebuilt moved
-   **10754 → 10777** for the swap. Exact f16 KV makes that gate moot — there is
-   no quantized V to pair with the drafter — but tumbler 1 pins the runtime on
-   its own: stock llama.cpp cannot load this file at any flag value. The 10754
-   tree stays on disk (`~/GitPackages/llama-prism/`) as the 27B rollback
-   runtime. `tests/frank_zealot_unit.bats` pins both paths.
-
-## Header census (read from the GGUF, not the card)
-
-`general.architecture qwen35`, `general.name Ternary-Bonsai-2-9B`,
-`general.size_label 9B`, `general.quantization_version 2`,
-`general.file_type 141`:
-
-| Key | Value | Note |
-|---|---|---|
-| `qwen35.block_count` | **33** | 32 trunk blocks + `blk.32` = the MTP block |
-| `qwen35.full_attention_interval` | 4 | ⇒ **8 full-attention + 24 Gated DeltaNet** trunk layers |
-| `qwen35.context_length` | 262,144 | native; the deployed slot equals it |
-| `qwen35.embedding_length` / `feed_forward_length` | 4,096 / 12,288 | |
-| `attention.head_count` / `head_count_kv` | 16 / **4** | |
-| `attention.key_length` / `value_length` | 256 / 256 | |
-| `ssm.conv_kernel` / `state_size` / `group_count` | 4 / 128 / 16 | the 27B: 4 / 128 / 16 |
-| `ssm.time_step_rank` / `inner_size` | **32 / 4,096** | the 27B: 48 / 6,144 |
-| `rope.freq_base` / `dimension_count` / `dimension_sections` | 1e7 / 64 / `[11,11,10,0]` | |
-| `qwen35.nextn_predict_layers` | **1** | the MTP head is IN the file |
-| `general.sampling.*` | temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / repeat 1.0 | the card recipe; llama.cpp reads it itself |
-
-**KV accounting.** Only the 8 full-attention layers scale with context:
-8 × 4 KV heads × 256 dim = **8,192 elements/token per cache** (the engine prints
-`n_embd_k_gqa = n_embd_v_gqa = 1024`). **The deployed pair is exact: f16 K +
-f16 V bills 32,768 B/token** (2 B/element × 8,192 elements × 2 caches), so the
-524,288-position pool is **16,384 MiB**. The q8_0 pair this endpoint shipped
-first billed **17,408 B/token** — 1.0625 B/element, 32 codes plus one 2-byte
-scale per 32-block — the **13,056.00 MiB** its ledger printed at the 786,432
-pool, exactly HALF the 27B's 34,816 B/token.
-
-**Corrected arithmetic, and it matters.** The note that lived here and in
-`frank-zealot.env` — "f16 would bill 26,112 MiB" — doubled the q8_0 figure
-*including* its scale overhead, overstating exact KV by 6.25 %. Exact f16 at the
-786,432 pool is **24,576 MiB** against the **24,123 MiB** the driver leaves
-addressable (`nvidia-smi`'s total column reads 24,576; used + free reads 24,123
-— the difference is driver reservation). The wall stands in both readings; the
-honest rate is what makes the fit calculable instead of superstitious. **Halving
-the POOL is what funds exact KV** — the window is already at the native ceiling
-and there is no YaRN in this path.
-
-The 24 DeltaNet layers bill a fixed recurrent state per SLOT instead: the 3-slot
-ledger printed `RS buffer size = 150.75 MiB (3 cells, 32 layers, 3 seqs)` ≈
-**50.25 MiB a cell**, so two slots bill 100.50. The SPEC-ARMED arms printed
-452.25 MiB — MTP depth 2 bills three seqs per slot, so the drafter triples even
-the recurrent state. Pool arithmetic alone under-prices the shape.
-
-## The shape (`scripts/frank-zealot.env`)
-
-`--ctx-size 524288 --parallel 2` (⇒ `n_ctx_slot = 262,144`), `--flash-attn on`,
-`--cache-type-k f16 --cache-type-v f16`, `--batch-size 4096`,
-`--ubatch-size 2048`, `--n-gpu-layers 99`, `--jinja`, `--reasoning auto`,
-`--reasoning-format deepseek`, `--metrics`, `--mmproj …`, `--log-verbosity 4`,
-`--presence-penalty 1.0`,
-**`SPEC_TYPE=none`** (which strips `--spec-type` AND every `--spec-draft-*`).
-
-Per-slot window is the **contract** the harness quotes: `models.yml` carries
-`contextWindow: 262144`, and it did **not** move with the slot count because
-llama.cpp divides `--ctx-size` across slots. It refuses a prompt longer than the
-SLOT with a 400 naming the limit — never a truncation — and since this swap the
-slot window equals the native window, so no YaRN and no cap is in play.
-`tests/frank_zealot_unit.bats` now gates the arithmetic: the shipped dtype ×
-pool plus the measured non-KV ledger terms must fit 24,123 MiB with ≥1 GiB of
-slack, and the gate is proven to REFUSE the impossible arm (exact KV at the old
-786,432 pool = 24,576 MiB of cache alone).
-
-## The MTP verdict: the head is in the file and it is still OFF
-
-The drafter is **good**: acceptance 0.798 / 0.617 per position, mean accepted
-length 2.41, and the card recommends depth 2. It is OFF because the draft
-context bills its own **f16** KV over the WHOLE pool — `blk.32` carries real
-attention (4 KV heads × 256 × 2 caches × 2 B = **4,096 B/position** at the
-default dtype). The draft dtype is itself a knob — `--spec-draft-type-k` /
-`--spec-draft-type-v`, both default f16, verified against this build's `--help`
-(so the "no flag exists" claim that lived here is FALSE) — and quantizing the
-draft to q8_0 would bill 2,176 B/position, **1,088 MiB** at the shipped pool.
-**No measured arm runs a drafter beside exact KV**; every row below is f16 draft.
-
-The census below is the **q8_0 era**: every row is q8_0 K + q8_0 V at the
-786,432 pool, where the draft bill was 3,072.00 MiB and 4,830 MiB with its
-compute buffers. Booted arms, same card, same traffic, 2026-10-18
-(`/completion` `timings`, greedy 512 tokens; never the first request after a
-boot):
-
-| slots | spec | ubatch | used / free | decode | 3-stream | prefill @106k |
-|---|---|---|---|---|---|---|
-| **3 × 262,144** | **off** | **2048** | **17,876 / 6,249** | **167.2** | **280.2** | **2,877** |
-| 3 × 262,144 | on | 2048 | 22,706 / 1,419 | 109.2 | — | — |
-| 3 × 262,144 | on | 1024 | 21,400 / 2,725 | 213.5 | 250.8 | 2,395 |
-| 2 × 262,144 | on | 2048 | 17,178 / 6,947 | 213.2 | — | — |
-| 1 × 262,144 | on | 2048 | 11,652 / 12,473 | 214.2 | — | — |
-
-**MTP is worth +28 % single-stream decode whenever the card has room** (rows
-4–5 prove the drafter itself is a win; row 2 proves the squeeze is what halves
-it). The arm that ran it at three windows was `SPEC_TYPE=draft-mtp` +
-`UBATCH_SIZE=1024` (row 3), paying **prefill −17 %** and **2,725 MiB of
-headroom** on a card whose mmproj worst-case lands AFTER the engine's own fit
-check.
-
-**Under exact KV an f16-draft arm is priced out of this card.** The draft
-context is built at the TARGET's pool, so at 524,288 positions it bills
-4,096 × 524,288 = **2,048.00 MiB** plus its own buffers, on a trunk that already
-projects **~21,200 MiB** of the 24,123 MiB addressable. The doors back to
-speculation are **one window per conversation** (row 5's class of shape), **KV
-back to q8_0** with `UBATCH_SIZE=1024`, or the **unmeasured**
-`--spec-draft-type-k q8_0 --spec-draft-type-v q8_0` arm — 1,088 MiB of draft,
-legal on b10777, never booted beside exact KV. Frank chose exact KV over the
-drafter on 2026-10-18, and the endpoint's actual job is what made that
-affordable: the fleet's **compaction hub** is prefill-bound on ~170k inputs,
-prefill is weight-bandwidth bound and does not re-read the cache per generated
-token — so the dtype is paid in decode and concurrency, not in the work this
-service exists to do.
-
-## Boot ledger — the q8_0 3-slot production boot (2026-10-18 13:19, under the unit, verbosity 4)
-
-```
-CUDA0 model buffer size  =  2036.71 MiB    # + 257.66 MiB CPU_Mapped (mmap)
-CUDA0 KV buffer size     = 13056.00 MiB    # q8_0 K + q8_0 V, 786,432 positions
-CUDA0 RS buffer size     =   150.75 MiB    # 3 cells, 32 layers, 3 seqs
-CUDA0 compute buffer     =  1409.02 MiB    # + 1088.87 MiB CUDA_Host, 2.84 output
-mmproj worst-case (est.) =   868.32 MiB    # 248.10 CUDA + 24.93 CPU, AFTER fit
-nvidia-smi settled: 17,922 MiB used / 6,203 MiB free of 24,123
-n_slots = 3, n_ctx_slot = 262144, kv_unified = false
-```
-
-No clamp line, no `cudaMalloc` error. The engine's own projection (21,439-class
-numbers from the 27B era) excludes the projector — **read settled `nvidia-smi`,
-never the projection**.
-
-**The exact-KV arm's projection, recorded before its own boot.** Every term of
-that ledger survives the reslice except the KV pool (13,056 → 16,384 MiB) and
-one recurrent-state cell (−50.25 MiB), so the pre-flight prediction is
-`17,922 − 13,056 + 16,384 − 50 ≈ 21,200 MiB used / ~2,920 free`, with ~2 GiB
-still standing through the mmproj worst case. **Replace this paragraph with the
-MEASURED ledger and settled `nvidia-smi` from the first f16 boot** — a
-projection is a pre-flight check, not a record, and the whole reason the shape
-changed dtype is that projections drift from ledgers.
-
-## Measured — the q8_0 3-slot boot (`/completion` `timings`, greedy, `ignore_eos`)
-
-- **168.7 t/s** decode — three runs of 512 tokens: 168.4 / 168.7 / 168.7
-  (the trial arm printed 167.2; same shape, same noise band).
-- **~300 t/s aggregate** across three concurrent 512-token streams
-  (102.8 t/s per stream, round 1 and 2 both) — the trial arm's 280.2.
-- **2,877 t/s prefill at 106,418 tokens** — the trial arm; the production boot
-  has not re-run that depth.
-- The 27B record it replaces: 76.6 decode, 109.4 aggregate across two streams,
-  691.7 prefill at 166k. Beaten on every axis.
-- The **first request after a boot is a warmup artifact** (37.7 t/s class) —
-  never report it. And never time a wall-clock aggregate on a generation that
-  stops on its own: a 24-token countdown measured 72 t/s across three streams
-  while the same shape with `ignore_eos` gave 300.
-
-**These are the numbers the exact-KV arm is judged against.** Exact KV makes
-attention read 1.882× the bytes per generated token, so a decode drop is
-EXPECTED and has to be measured, not assumed; prefill should hold. Re-run the
-same instrument on the first f16 boot — greedy, `ignore_eos`, 512 tokens, the
-second request or later, and the two-stream aggregate in place of the
-three-stream one — and record it here.
-
-## Contract traps
-
-- **`--metrics` is mandatory.** Without it llama.cpp answers **501** on
-  `/metrics`, so the Prometheus scrape reads DOWN while `/health` answers 200 —
-  that is how this endpoint sat red across four payload generations
-  (`up{job="frank-zealot"}` oldest retained sample, 2026-10-11, is already 0).
-- **Q8_0 mmproj, not BF16.** The publisher's card: the BF16 tower "can crash
-  llama-server when several image requests run at once; Q8_0 is not affected" —
-  and `--parallel 2` is still precisely that shape.
-- **Only the `image_url` content part reaches the model.** The legacy top-level
-  `images` field bumps `usage.prompt_tokens` and the model still invents an
-  answer. Probe with a solid color and expect the exact word.
-- **≥256 px probe images** or the model answers "video".
-- **Thinking tokens count against `max_tokens`.** A vision probe at 64 tokens
-  returns EMPTY `content` + `finish_reason: "length"` — the endpoint looks blind
-  while it is only mid-thought. Budget ≥512.
-- **`presence_penalty` has no GGUF metadata key**, so the one recipe value that
-  must reach the argv is this one. The serve script uses
-  `${PRESENCE_PENALTY=1.0}` — **not `:=`** — because `""` is the documented
-  opt-out and `:=` would silently re-add the flag (caught by the seam, fixed
-  2026-10-18).
-- **Spec flags move as a SET.** A bare `--model-draft`/`--spec-type` arm without
-  the whole set has segfaulted at load on this fork family in the past;
-  `SPEC_TYPE=none` strips all of them.
-- **Base alignment is the censored base lineage, and the CRACK swap is NOT
-  one line here.** `/home/dank/models/Bonsai2-CRACK/` matches the **27B**
-  geometry, not this 9B file; a dealigned 9B would have to be fetched.
-- **A raw `/completion` request** logs `failed to parse reasoning content
-  (… parse_reasoning=true)` when the model closes a bare prompt with `` —
-  cosmetic (`probe_request.cpp:789`); `/v1/chat/completions` parses
-  `reasoning_content` correctly. Do not lower `VERBOSITY`: the KV/buffer ledger
-  is what VRAM forensics read.
-- **A failed stop leaves a `failed` unit, and an orphan keeps answering.** On
-  2026-10-18 the trial stop hit `State 'stop-sigterm' timed out. Killing.` →
-  `Active: failed (Result: timeout)`, while the hand-booted arm kept serving
-  7771 — a live port with no unit behind it. `TimeoutStopSec=180` is now on the
-  unit (llama.cpp needs the drain time to release slots and the card); the
-  hand-booted arm must be killed by PID, never `pkill -f llama-server`.
-- **A leftover arm drop-in hijacks the production boot, and no health check
-  sees it.** The sanctioned bench method is a transient
-  `Environment=FRANK_ZEALOT_ENV_FILE=/tmp/<arm>.env` drop-in, and an arm env
-  `source`s the repo env and then overrides it. On 2026-10-18 a stale
-  `frank-zealot.service.d/ptq1.conf` — written 03:29 for the 27B **PTQ1_0** arm
-  Frank had ordered at 02:36 and measured from 03:14 (deploy ledger below) —
-  was still installed, so the cutover deploy restarted
-  the unit and booted THAT: `--model …/Bonsai2-PTQ1_0/Ternary-Bonsai-2-27B-PTQ1_0.gguf
-  --ctx-size 401408 --parallel 3 --batch-size 3072 --ubatch-size 1536`, three
-  slots of 133,888, 21,188 MiB used. `/health` was 200, `/v1/models` said
-  `frank-zealot` (the alias is a flag, not a fact), and the Playwright gate
-  passed 33/33 — none of those read the payload. **After deploying an endpoint
-  that was recently benched: `systemctl show -p DropInPaths`, then
-  `tr '\0' ' ' </proc/$pid/cmdline`, then the ledger — in that order.** Retire
-  the drop-in to `~/.AGENTTRASH/<date>-<topic>/`, `daemon-reload`, restart.
-
-## Compaction hub
-
-`models.yml` routes `compactionModel` for `frank-archon`, `frank-dragoon` and
-`frank-zealot` here — 262,144 is the widest window in the fleet and the only one
-that comfortably fits a worst-case compaction input (~170k tokens; the clamp is
-`min(maxTokens 16384, contextWindow − est − 4096)`, compaction fires at 85 % of
-the window ≈ 222,822 est, worst case ≈ 245,891 ≤ 262,144). What the reslice to
-**two** slots does cost is here: a compaction request still runs beside ONE live
-generation, but with two conversations already busy a third waits — the Gemma4
-era's `--parallel 1` serialization law at half strength. The window itself is
-untouched, and the window is the term the compaction contract depends on.
-
-## Deploy ledger (2026-10-18)
-
-- `./scripts/deploy.sh dankclaw --service frank-zealot` at 13:16 PDT; the e2e
-  gate ran 33/33. The first boot was the hijacked arm above; the drop-in went
-  to `~/.AGENTTRASH/20261018-zealot-ptq1-arm/` and the 13:19 restart is the
-  ledger recorded here.
-- Live probes on the shipped boot: chat `27*59` → **1593**; `where_am_i` tool
-  call fired with schema-correct empty arguments; 512 px solid-red PNG →
-  **Red** at 278 prompt tokens with `process_mtmd` chunk encoding in the
-  journal; `/props` reports `n_ctx 262144` and `presence_penalty 1.0`;
-  `/metrics` answers 200 with 15 `llamacpp:*` families and
-  `up{job="frank-zealot"}` is **1** in Prometheus for the first time in its
-  retained history.
-- OpenWebUI picker rows re-keyed in the live DB (`Connection.backup()` snapshot
-  `webui.db.bak-20261018` taken first; `frank-dragoon`'s `context.length` also
-  corrected 262,144 → 200,704 — see [[OpenWebUI]]).
-- **What the bench window left behind is not garbage — the PTQ1_0 arm was
-  ordered, measured, and then scratched by Frank himself.** The instruction at
-  **02:36** was: run frank-zealot on `Ternary-Bonsai-2-27B-PTQ1_0.gguf`, the
-  **Q8** mmproj, batch/ubatch **3072/1536**, and as much context as fits in
-  **two** contexts. The unit booted it at **03:14** (journal:
-  `type ptq1_0: 402 tensors`, `file type = PTQ1_0 - 1.75 bpw ternary`, mmproj
-  loaded, 2 × 200,704) and served 24 minutes of traffic. Its own
-  `print_timing` lines measure **~48 t/s** single-stream decode
-  (48.0 / 47.9 / 47.6 / 49.0 t/s on 512 / 229 / 512 / 451-token generations),
-  25–27 t/s per stream with two streams live, and prefill **1,295 t/s at
-  30,637 → 943 at 97,856 → 801 t/s at 141,365** tokens. It bought the
-  1,201 MiB the fetch script promised and paid for it on decode, exactly as the
-  publisher's Ampere row warned (the PQ2_0 record for the same weights is
-  76.6 t/s). At **03:33** Frank scratched the whole 27B plan and ordered the
-  9B PQ2_0 + MTP shape deployed here; the arm process ended at 03:38.
-- The actual failure was **documentation, not measurement.** An approved
-  payload that survives only as a systemd drop-in plus a journal window is,
-  a day later, indistinguishable from a rogue arm — which is why the 13:16 boot
-  read as a hijack. Payload decisions belong in this runbook the moment they
-  are made. The evidence is still perishable: `/tmp/zealot-arm-current.env` and
-  the 9B `/tmp/arm-*.jsonl` results die on reboot, and only the journal
-  (`journalctl -u frank-zealot --since "2026-10-18 03:10"`) holds the PTQ1_0
-  numbers. `/home/dank/models/Bonsai2-PTQ1_0/` (5,946,648,928 B + hardlinked
-  mmproj) stays on disk as the 27B rollback.
-
-## Fallback rungs
-
-Cheapest first; every one is a single line of `scripts/frank-zealot.env`, and
-the KV figures are for the 524,288-position pool unless stated otherwise:
-
-1. `UBATCH_SIZE=1024` — hands back **1,306 MiB** of compute buffers measured on
-   this card (prefill −17 %).
-2. `CACHE_TYPE_V=q8_0` — exact K, quantized V: the pool goes 16,384 →
-   **12,544 MiB**, freeing 3,840, and it is the halfway door back.
-   `--flash-attn on` is what keeps quantized V legal.
-3. `CTX_SIZE=401408` — two 200,704 conversations: exact KV at
-   **12,544.00 MiB**, same 3,840 freed, but `models.yml` `contextWindow` MUST
-   follow the per-slot number or long turns start 400-ing.
-4. Revert to `q8_0` K + `q8_0` V at `CTX_SIZE=786432 PARALLEL=3` — the shape
-   this one replaced: 13,056.00 MiB of cache and the THIRD conversation back.
-5. `CACHE_TYPE_K/V=q4_0` — the far rung: 10,240 B/token ⇒ **5,120 MiB** here
-   (the 27B era measured q4_0 at 16,392 used / 7,733 free at 2 × 200,704).
-
-Any retune has to keep the seam's fit gate green — it is the only instrument
-that can see a boot that would die at `cudaMalloc`.
-
-## Rollback
-
-Revert the swap commit. On disk: the **27B payload**
-(`/home/dank/models/Bonsai2-PQ2_0/`) + the **10754** fork tree, the Qwen3.5-9B
-Q6_K payload (`/home/dank/models/Qwen35-9B-HauhauCS-Q6K/`) + master CUDA build,
-the Gemma4, MiMo, Swift and Ornith payloads. Then
-`./scripts/deploy.sh dankclaw --service frank-zealot`.
-
-## Related
-
-- [[frank-zealot-llama-prism-bonsai2-pq20]] — the retired 27B era (rollback target)
-- [[Llama Cpp Serving Sizing]] — the two-boot VRAM pricing method
-- [[frank-dragoon-flash-server-v2-native]] — the other 2026-10-18 cutover
-- [[deployment]] — deploy flow and incidents

serverAppConfigs-docs/LLM/frank-zealot-llama-qwen35-9b-q6k-2x200704.md

@@ -1,167 +0,0 @@
----
-type: reference
-title: "frank-zealot on Dankclaw — llama.cpp, Qwen3.5-9B Q6_K, 2 x 200704, f16 KV (RETIRED, rollback target)"
-description: "The frank-zealot runbook of 2026-10-07 → 2026-10-17 (RETIRED — rollback target) — llama.cpp master (CUDA sm_86) serving HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K on the RTX 3090: two conversations of 200,704 tokens with exact f16 KV, vision through the publisher's BF16 projector, MTP impossible (no tensors in the file), and the measured geometry that priced the shape."
-tags:
-  - llm
-  - frank-zealot
-  - dankclaw
-  - llama-cpp
-  - gguf
-  - rtx3090
-timestamp: 2026-10-07
----
-
-# frank-zealot — llama.cpp Qwen3.5-9B Q6_K, 2 × 200,704, f16 KV
-
-**RETIRED 2026-10-07 → 2026-10-17** (AGENTS.md Runbook Index), superseded by
-[[frank-zealot-llama-prism-bonsai2-pq20]] (27B PQ2_0, 2 × 200,704 q8_0 KV, build
-10754), itself retired 2026-10-17 → 10-18 and now the rollback target: :7771
-today is llama.cpp PrismML build **10777** on
-`Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, 2 × 262,144 exact f16 KV, MTP off by
-measurement (AGENTS.md Runbook Index active row; `scripts/frank-zealot.env`).
-Transient 2026-10-18: :7771 is inside a bench window — the unit is `failed`
-since 03:14 and a hand-booted process answers the port; restore with
-`./scripts/deploy.sh dankclaw --service frank-zealot`.
-This era is the rollback target: the payload is still on disk
-(`stat -c %s /home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf`
-→ 7359259008, read 2026-10-18) with its master CUDA build. Same census: the
-10754 rollback payload `Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-PQ2_0.gguf` is
-present at 7206168928 with its Q8_0 mmproj at 629246976, and
-`Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf` at 2675410720 —
-nothing this era or its successors need has been purged. It replaced the
-[[frank-zealot-llama-gemma4-q4km-200704]] era (Q4_K_M on ONE 200,704
-slot). The swap is a capacity trade: the 9B dense-hybrid rung costs ~7 GiB of
-weights where the 26B MoE cost ~16.6 GiB, and the freed room buys the second
-conversation slot and unquantized KV back.
-
-## Identity
-
-| | |
-|---|---|
-| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
-| Endpoint | `http://192.168.0.143:7771` — served model id **`frank-zealot`** (`ALIAS` in the env; the SERVICE keeps the `frank-zealot` name), `/v1` OpenAI surface |
-| Runtime | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` — `--version` on the disk build reads `0.5.0-dev (build 1, commit 5fc4f3c)` (built 2026-10-01); its `CMakeCache.txt` says `GGML_CUDA_FA_ALL_QUANTS:BOOL=OFF` and that tree carries no CUDA-side guard for the option (`ggml/src/ggml-cuda/fattn.cu:438-449` dispatches Q4/Q5 KV unconditionally), so quantized KV never required it; `qwen35` is registered in the build's own `src/llama-arch.cpp` and `src/models/qwen35.cpp` exists at that commit |
-| Unit | root systemd `frank-zealot.service` ([`scripts/frank-zealot.service`](../../scripts/frank-zealot.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
-| Serve script | [`scripts/serve-frank-zealot.sh`](../../scripts/serve-frank-zealot.sh), tunables [`scripts/frank-zealot.env`](../../scripts/frank-zealot.env) |
-| Weights | `/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf` — HauhauCS **Q6_K**, 7,359,259,008 B, sha256 `c0ba7beb68fd3fe47891bd549486d38dcf62d00817296ea314ad37017f5a4986` |
-| Projector | `mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf`, 921,704,448 B, sha256 `05f662501f8bd45607b079723a3e238a4e888fd085a10a53f4057a0e250f6934` — the model is natively multimodal and the Q6_K file carries no vision tensors, so this file IS the vision capability |
-| Fetch | [`scripts/download-frank-zealot.sh`](../../scripts/download-frank-zealot.sh) — `-f`, byte-count + sha256 verified (the hashes are computed locally: the repo is Xet-backed, so the Hub publishes file OIDs, not file sha256), idempotent, `--verify-only` |
-| Seam | `tests/frank_zealot_unit.bats` (argv assertions against a stub engine) |
-
-## Payload geometry (read from the GGUF header, not the model card)
-
-Range-read of the file's own metadata + tensor-info section:
-
-|Key|Value|
-|---|---|
-|`general.architecture`|`qwen35` (dense hybrid: Gated DeltaNet + softmax attention)|
-|`qwen35.block_count`|32, `full_attention_interval` 4 ⇒ **8 full-attention layers (3, 7, 11, …, 31) + 24 DeltaNet layers**|
-|attention|16 heads, **4 KV heads**, `key_length` = `value_length` = 256|
-|`qwen35.ssm.*`|`conv_kernel` 4, `state_size` 128, `group_count` 16, `time_step_rank` 32, `inner_size` 4096|
-|context / vocab|262,144 native / 248,320|
-|tensor types|Q6_K × 250 + F32 × 177 — standard types only, so the stock master build loads it (the ROCmFPX-style custom-quant lock-in does not apply)|
-|MTP|**no `nextn`/`mtp` tensors exist** — in-file MTP is impossible; the card's "MTP support" line describes the base architecture, not this quantized export|
-|chat template|supports `enable_thinking`, `tools`/`tool_call`, image/video content parts; it has **no `reasoning_effort` variable** (unused Jinja variables are ignored, so the harness's effort map is inert here, not an error source)|
-
-## Shape
-
-`--ctx-size 401408 --parallel 2` — llama.cpp divides the pool across slots,
-so each conversation gets `n_ctx_slot = 200,704` (Frank's cap; the harness
-`contextWindow` quotes the per-slot number). `--flash-attn on`,
-**`--cache-type-k f16 --cache-type-v f16`** (exact KV), batch 4096 /
-**ubatch 2048**, `--log-verbosity 4`, `--jinja --reasoning auto
---reasoning-format deepseek`, `--mmproj`, **speculative decoding OFF**
-(`SPEC_TYPE=none` strips both spec flags — the file has no MTP tensors).
-
-**KV rate law.** The pool scales with the 8 full-attention layers: 8 × 4 KV
-heads × 256 dim = 8,192 elements/token per cache, so f16 K + f16 V bills
-32,768 B/token = **32 KiB/token** → 12.25 GiB at the 401,408 pool. The 24
-DeltaNet layers bill a fixed recurrent state **per slot**, not per token —
-that is what makes two 200k slots affordable where the 26B MoE fit only one.
-
-**Priced from boots, not formulas.** MiMo-V2.6-Distill-Qwen-9B (identical
-qwen35 geometry, 7,793,710,624 B, same mmproj class) at 2 × 200,704 f16 K +
-f16 V measured **21,276 used / 2,849 free** at ubatch 1024 and **21,846 /
-2,279** at ubatch 2048 — see [[frank-zealot-llama-mimo-q6k-2x200704]].
-This file is 415 MiB lighter in weights and its projector is 3 MiB heavier,
-so the projection is ~21,430 used / ~2,700 free.
-
-Ubatch 2048 is the measured CUDA optimum for this card and arch (+4.6 %
-prefill at 8.9k depth, decode flat, cost 616 MiB); `1024` is the documented
-fallback rung if a future shape lands under the 2 GiB floor.
-
-## Measured boot (2026-10-07, deploy `ec6184d`)
-
-`nvidia-smi`: **21,436 MiB used / 2,689 MiB free** — the MiMo-priced
-projection (~21,430 / ~2,700) was within 11 MiB. Engine ledger
-(`--log-verbosity 4`, `journalctl -u frank-zealot`):
-
-|Line|Value|
-|---|---|
-|`llama_context`|`n_ctx = 401408`, `n_ctx_seq = 200704`, `kv_unified = false`|
-|`llama_kv_cache`|`CUDA0 KV buffer size = 12544.00 MiB` — `size = 12544.00 MiB (200704 cells, 8 layers, 2/2 seqs), K (f16): 6272.00 MiB, V (f16): 6272.00 MiB`|
-|`sched_reserve`|`CUDA0 compute buffer size = 1136.06 MiB`, `CUDA_Host compute buffer size = 848.07 MiB`|
-|`clip_model_loader` (mmproj)|`model size: 878.99 MiB` + `reserve_compute_meta: CUDA0 248.10 MiB, CPU 24.93 MiB`|
-|`srv load_model`|`n_slots = 2, n_ctx_slot = 200704`|
-|`print_info`|`n_ctx_train = 262144`, `n_head_kv = 4`|
-
-The `32 KiB/token` law is confirmed exactly: 401,408 × 32 KiB = 12,544 MiB,
-which is what the engine printed. The `n_ctx_seq (200704) < n_ctx_train
-(262144)` line is informational (a context **below** the trained window is
-safe; the Gemma4 era's warning was the opposite case, a drafter running 53 %
-**past** its window). Weights load in ~6 s from page cache; `/health` answers
-`{"status":"ok"}` and `/v1/models` serves the alias `frank-zealot`.
-
-**Wire rename (2026-10-09):** `ALIAS` moved `frank-secondary` → `frank-zealot`
-(commit `7c77326`; the service-file rename to the `frank-zealot.*` names
-followed 2026-10-10, commit `55f87fc`),
-so the served id matches the harness model id and the family-facing name.
-Consumers re-keyed in the same change: OpenWebUI (`model` row, `access_grant`
-row, `openai.api_configs` index-2 `model_ids`) and `models.yml` (already
-renamed 2026-10-08). llama.cpp does NOT validate the requested model name —
-it answers and echoes the served alias — so a stale client keeps working;
-the picker and any id-keyed grant is what breaks. See [[OpenWebUI]].
-
-The deploy's Playwright gate passed (33 tests) and the fleet e2e set is
-unchanged.
-
-## Client contract
-
-llama.cpp surface: model id `frank-zealot`, native tools via `--jinja`,
-thinking in **`reasoning_content`**, vision via OpenAI `image_url` content
-parts (probe with a ≥ 256 px image, or the Qwen-VL tower answers "video").
-The publisher's recipe ships in `models.yml` `extraBody`: thinking mode
-`temperature 0.6 / top_p 0.95 / top_k 20 / min_p 0` (non-thinking mode is
-0.7/0.8/20/0). The card's advice "maintain at least 128K context to preserve
-thinking" is satisfied by the 200,704 slot.
-
-**Compaction hub**: `models.yml` routes `compactionModel` for all three
-endpoints here — still true 2026-10-18 (`~/.omp/agent/models.yml:25,107,185` all name `llamacpp (dankclaw-3090)/frank-zealot`). With `--parallel 2` a compaction request no longer serializes
-behind a live generation (the Gemma4 era's `--parallel 1` law), and the
-200,704 slot still fits the worst-case compaction input (~133k tokens).
-
-## Rollback
-
-|Target|How|
-|---|---|
-|Gemma4-26B-A4B Q4_K_M, 1 × 200,704, f16 K + q8_0 V|revert the swap commit — payload stays at `/home/dank/models/Gemma4-26B-A4B/`; see [[frank-zealot-llama-gemma4-q4km-200704]]|
-|MiMo-V2.6 Q6_K, 2 × 200,704, f16 KV|`MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + its mmproj — measured 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048|
-|Swift-1.5-27B IQ3_XXS 2 × 114,688 + MTP|payload on disk at `/home/dank/models/Swift15-UncIQ3XXS/`|
-
-## When it misbehaves
-
-Same decision tree as the predecessors: `No CUDA GPUs are available` ⇒
-`nvidia-modprobe -u -c=0`; a 400 naming 200,704 ⇒ the conversation outgrew
-one slot, not the pool; queue behind two long generations ⇒ the slot ceiling
-is two, by contract; a boot that OOMs ⇒ read the verbosity-4 ledger in
-`journalctl -u frank-zealot` (KV MiB, compute buffers) before touching
-flags — the engine prints the exact figures the shape needs.
-
-**`status=203/EXEC` boot loop**: the rewrite of `serve-frank-zealot.sh`
-shipped without its `#!/usr/bin/env bash` line, and systemd failed the
-`ExecStart` before the script ran — 10 s restarts, `Mem peak: 1.9M`, and
-`nvidia-smi` flat at 1 MiB used (the GPU is never touched). Journal signature:
-`Failed at step EXEC spawning /home/dank/…/serve-frank-zealot.sh`. A
-shebang is not decoration on this host: every LLM serve script is a direct
-`ExecStart`, so `head -1` is part of the deploy check (commit `ec6184d`).

serverAppConfigs-docs/LLM/frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688.md

@@ -1,353 +0,0 @@
----
-type: reference
-title: "frank-zealot on Dankclaw — llama.cpp, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 x 114688, draft-mtp, q8_0 KV"
-description: "The frank-zealot runbook of 2026-10-05 → 2026-10-07 (RETIRED — rollback target; live from 2026-10-05 evening, renamed from frank-small 2026-10-06, per-slot lowered to 112k and ubatch raised to 1024 the same evening) — llama.cpp master (CUDA sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS (MTP head in the file, drafted) on the RTX 3090: TWO 114,688-token conversations at Frank's 112k cap, q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the publisher's own BF16 projector, and the full two-slot + drafter arithmetic."
-tags:
-  - llm
-  - frank-zealot
-  - dankclaw
-  - llama-cpp
-  - gguf
-  - rtx3090
-  - qwen35
-  - mtp
-  - vision
-timestamp: 2026-10-06
----
-
-# frank-zealot — llama.cpp Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 × 114,688, draft-mtp, q8_0 KV
-
-> **RETIRED 2026-10-06 → 2026-10-07** (AGENTS.md Runbook Index; cut over
-> 2026-10-05 ~23:26, reshaped three times on 2026-10-06, renamed from
-> frank-small the same night). Superseded by
-> [[frank-zealot-llama-qwen35-9b-q6k-2x200704]]; the production shape on :7771
-> today is llama.cpp PrismML build **10777** on
-> `Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, 2 × 262,144 exact f16 KV, MTP off by
-> measurement (AGENTS.md Runbook Index active row); the 27B PQ2_0 / 2 × 200,704
-> shape of [[frank-zealot-llama-prism-bonsai2-pq20]] is that era's rollback
-> target. Transient 2026-10-18 bench window on :7771 — the unit is `failed`
-> since 03:14 and a hand-booted process answers the port; restore with
-> `./scripts/deploy.sh dankclaw --service frank-zealot`).
-> **Rollback target**: the payload is still on disk —
-> `stat -c %s` on 2026-10-18 gives 10442827392 for
-> `/home/dank/models/Swift15-UncIQ3XXS/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf`
-> and 931145888 for its `mmproj-BF16.gguf`. A 27B replaces
-> the 9B on the same 24 GiB card: KV at FP8 class, vision on. The
-> morning order (drop MTP, ubatch 512, TWO 128k contexts) bought the
-> second slot — 1,397 MiB free. The evening order swapped to the
-> UNCELERSED Swift-1.5 at RCO-IQ3_XXS and put MTP back ON — landing
-> **489 MiB free**: the MTP draft ctx costs **~2.4 GiB, not the ~0.4
-> GiB assumed** (the drafter's KV is built against the target's full
-> context, not the 3-token proposal depth), paid for by the lighter
-> quant (−1.56 GiB). The late order dropped the per-slot window
-> 131,072 → **114,688** (Frank's 112k cap) and climbed the prefill ladder
-> ubatch 512 → 1024 → **2048** — landing **1,005 MiB free** (1,751 at
-> the 1024 rung; the 2048 rung cost 746 MiB, ~130 over the 616 MiB
-> measured prediction). This supersedes BOTH earlier 2026-10-05
-> payloads: the live MiMo-V2.6 Q6_K shape
-> ([[frank-zealot-llama-mimo-q6k-2x200704]], now retired) and the
-> Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but
-> never deployed.
-
-## Identity
-
-| | |
-|---|---|
-| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
-| Endpoint | `http://192.168.0.143:7771` — alias `frank-secondary` on the wire until the 2026-10-09 rename to `frank-zealot` (commit `7c77326`), `/v1` OpenAI surface |
-| Runtime | llama.cpp master (0.5.0-dev, commit 5fc4f3c), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (quantized V needs `--flash-attn on`; it does NOT need `GGML_CUDA_FA_ALL_QUANTS` — see the KV section) |
-| Unit | root systemd `frank-zealot.service` ([`scripts/frank-zealot.service`](../../scripts/frank-zealot.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
-| Serve script | [`scripts/serve-frank-zealot.sh`](../../scripts/serve-frank-zealot.sh), tunables [`scripts/frank-zealot.env`](../../scripts/frank-zealot.env) |
-| Weights | `/home/dank/models/Swift15-UncIQ3XXS/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf` — ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF **RCO-IQ3_XXS -mtp**, 10,442,827,392 B, sha256 `e13d3e74…` (qwen35 hybrid: 64 real layers, 16 full-attention + 48 Gated DeltaNet, native 262,144; **MTP head in the file**: blk.64.nextn.*, `nextn_predict_layers=1`, and DRAFTED with it) |
-| Projector | `…/Swift15-UncIQ3XXS/mmproj-BF16.gguf` — PUBLISHER-SHIPPED in the same repo, 931,145,888 B, sha256 `19acd7fc…` (27-layer ViT → 5120) |
-| Fetch | [`scripts/download-frank-zealot.sh`](../../scripts/download-frank-zealot.sh) — `-f`, byte count + sha256 verified per file (Xet repo: the Hub APIs expose no LFS oid, so the shas were computed locally after the first verified fetch; byte counts match the Hub's declared sizes) |
-| Seam | `tests/frank_zealot_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, spec pair (both flags present), batch pair, projector opt-out) |
-
-## Why this payload (2026-10-06 evening)
-
-Frank asked for the UNCELERSED Swift-1.5 with MTP back on. The RCO-IQ3_XXS
-is 1.56 GiB lighter than the IQ3_S-mtp it replaced — and that lighter
-weight is exactly what funds the MTP draft context (~2.4 GiB measured)
-PLUS the second 131,072 slot, while keeping the q8_0 KV contract and the
-two-slot shape Frank ordered that morning.
-
-**Uncensored, intentionally:** the publisher reports 23/100 refusals vs
-Swift 1.5's 98/100 (KL 0.0884 against the censored base). The refusal
-escape hatch is the same as every payload here: revert and redeploy.
-
-## Shape and KV contract
-
-`--ctx-size 229376 --parallel 2` — TWO conversations at Frank's 112k
-cap (2026-10-06 late: cut from the 128k the morning/evening shapes ran
-at, to fund ubatch 1024). `CTX_SIZE` stays divisible by `PARALLEL`; the
-seam enforces it, and the per-slot number (`229376 / 2 = 114,688`) is
-the product contract — models.yml follows at 114,688 (a longer prompt
-is a 400, never a truncation).
-
-**KV at FP8 class: `q8_0` K + `q8_0` V (Frank, 2026-10-05 evening).**
-llama.cpp has no true fp8 KV dtype; q8_0 is the 8-bit class (1.0625 B/elem
-⇒ 8.5 effective bits) and the only FP8-class lever this runtime moves.
-The order supersedes the same-day f16/f16 rule, which was written for the
-Ornith 9B fit — at 64 KiB/token f16 the single 131,072 slot alone would
-cost 8.0 GiB of KV. At q8_0 the rate is **34 KiB/token** (16 attn layers ×
-4 KV heads × 256 dim × 1.0625 B, K and again V). `--flash-attn on` is
-pinned: quantized V requires it. The era note claiming the build carried
-`GGML_CUDA_FA_ALL_QUANTS=ON` does not describe the binary on disk: its
-`CMakeCache.txt` reads `GGML_CUDA_FA_ALL_QUANTS:BOOL=OFF` (binary built
-2026-10-01 22:35) and the option is vestigial in that tree — `ggml/src/ggml-cuda/fattn.cu:438-449`
-dispatches the Q4/Q5 KV cases unconditionally.
-
-**Two slots + drafter, by arithmetic and by order (2026-10-06 morning /
-evening, the 2 × 131,072 era).** The morning order bought the second slot
-by spending the drafter and ubatch headroom (1,397 MiB free). The evening
-swap put the drafter BACK and changed the arithmetic: the draft ctx is
-built against the target's FULL context (262,144 — the draft window, not
-the 3-token proposal depth), so it costs **~2.4 GiB measured** (23,636 −
-22,728 + 1,561), not the ~0.4 GiB the one-slot era assumed. The lighter
-quant pays for it: **23,636 used / 489 MiB free** (nvidia-smi `free`
-field) — the tightest fit in frank-zealot's history, the 2–3 GiB floor
-knowingly violated, and the 512×512 vision transient (34 MiB) still fits
-and recovers.
-
-**The late reshape (2 × 114,688, ubatch 2048) is the current fit.** The
-per-slot cut frees 2 × 16,384 tokens × 34 KiB ≈ 1.05 GiB of KV; the
-ubatch rungs 512 → 1024 → 2048 spend it back on compute buffers (a
-prefill knob — decode never moves with it; the rungs cost ~0.3 and
-~0.75 GiB measured). Measured boot at 2048: **23,120 used / 1,005 MiB
-free**. The drafter's own KV shrank with its target (draft window
-229,376). Compute buffers allocate at graph capture during BOOT, so a
-miss is loud and fast — the failure mode is a crash loop, not silent
-corruption.
-
-## Speculative decoding — ON since 2026-10-06 evening (draft-mtp, depth 3)
-
-**Frank's evening order: MTP back on.** The MTP head ships IN the file —
-the `-mtp` GGUF carries `blk.64.nextn.*` (`qwen35.nextn_predict_layers=1`)
-— and `--spec-type draft-mtp` loads it against the main model's own
-weights (no `--model-draft`, no second draft model — two context-backed
-drafters cannot share a process). `SPEC_DRAFT_N_MAX=3` is the proposal
-depth. **ngram-mod stays skipped, not swapped in:** on a qwen35 hybrid,
-ngram speculation logs *"recurrent state rollback is not compatible with
-'ngram-mod' — disabling rollback support"* (fleet lesson from
-frank-dragoon's chain measurement, 2026-10-03). A rejected draft on Gated
-DeltaNet layers cannot be rewound by KV truncation — without the rollback
-checkpoints the DeltaNet state carries phantom transitions and output
-degrades silently. The flags move as a set: the serve script emits BOTH
-`--spec-type` and `--spec-draft-n-max` together or neither
-(`SPEC_TYPE=none` strips both).
-
-## Vision contract
-
-`--mmproj` is the entire vision path (the GGUF carries no vision tensors).
-Probes: a solid-color PNG **≥ 256 px** answers its exact color — a tiny
-probe undersupplies image tokens and the model invents ("video"); the
-legacy top-level `images` field is IGNORED, only the OpenAI `image_url`
-content part reaches the model. **New on this payload: the uncensored
-model THINKS more than the censored Swift 1.5** — a 64-token budget was
-eaten entirely by `reasoning_content` (`finish_reason: length`, empty
-`content`); 128 tokens clears it. Token budgets for vision probes must
-leave room for thinking.
-
-## Measured: the 1 × 131,072 MTP boot (2026-10-05, historical)
-
-- **Boot ledger**: `creating MTP draft context against the target model
-  '…/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf'`, `loaded multimodal
-  model, '…/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf'`, `n_slots = 1,
-  n_ctx_slot = 131072` — every expected line, no q8→f16 fallback.
-- **VRAM: 19,404 used / 4,721 MiB free** (nvidia-smi `free` field). The
-  4.5 GiB prediction held; f16 KV would have left ~700 MiB, so this free
-  number doubles as the q8_0 proof.
-- **Decode + MTP acceptance** (draft counts from the live `timings`):
-  temp 0.8 → 66.1 tok/s, 182/349 accepted (52.1 %); temp 0 → 66.9 tok/s,
-  166/318; **deployed recipe (temp 1.0/top_p 0.95/top_k 20) → 58.9 tok/s,
-  159/355 (44.8 %)** — temperature buys fluency, costs acceptance and
-  rate. For scale: MiMo Q6_K measured 94.4 tok/s on the same card — this
-  27B at IQ3_S runs ~2/3 the rate with ~3× the capability class.
-- **Vision**: 512×512 solid-red PNG → `Red`, `finish_reason: stop`.
-- **Template gate**: `reasoning_effort: xhigh` → 200 with
-  `reasoning_content` populated; `high` → HTTP 500 (Jinja raise) — the
-  models.yml fold is mandatory, exactly as the MiMo eras found.
-- Streaming completion: 200. No first-request silence window (the
-  halogen-era 45–60 s stall has no analogue here).
-
-## Measured: the 2 × 131,072 no-spec boot (2026-10-06 morning, historical)
-
-- **Boot ledger**: `loaded multimodal model, '…/Swift15-IQ3S/
-  mmproj-Qwen3.8-27B-F16.gguf'` and NO draft-context line (speculation
-  off). `/slots`: **2 slots, n_ctx 131,072 each** (pool 262,144).
-- **VRAM: 22,728 used / 1,397 MiB free** (nvidia-smi `free` field) — the
-  predicted 0.3–1.3 GiB window held; the old 2-3 GiB floor is knowingly
-  violated by Frank's order. After a 512×512 image probe: 1,363 MiB —
-  the vision transients fit.
-- **Decode without the drafter**: 48.0 tok/s (temp 0) / 47.8 (deployed
-  recipe temp 1.0). `draft_n` absent from `timings` (no drafter). The MTP
-  era's 58.9–66.9 tok/s is the honest price of the second context:
-  −18–28 %.
-- **Vision**: 512×512 solid-red PNG → `Red` again on this shape.
-- Deploy: commit 812d9eb, e2e gate 33/33 green on the live fleet.
-
-## Measured: the 2 × 131,072 draft-mtp boot (2026-10-06 evening, historical)
-
-- **Boot ledger**: `creating MTP draft context against the target model
-  '…/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf'`,
-  `loaded multimodal model, '…/Swift15-UncIQ3XXS/mmproj-BF16.gguf'`,
-  `n_slots = 2, n_ctx_slot = 131072` — every expected line, no q8→f16
-  fallback.
-- **VRAM: 23,636 used / 489 MiB free** (nvidia-smi `free` field) — the
-  draft ctx measured ~2.4 GiB, ~6× the ~0.4 GiB the one-slot era
-  assumed; the lighter quant (−1.56 GiB) plus the morning's no-spec base
-  still fund both slots. After a 512×512 image probe: 489 MiB — the vision
-  transient (34 MiB) fits and recovers.
-- **Decode + MTP acceptance** (draft counts from the live `timings`,
-  512-token essay): temp 0 → **72.9 tok/s**, 333/534 accepted (62.4 %);
-  deployed recipe (temp 1.0/top_p 0.95/top_k 20) → **63.3 tok/s**,
-  305/612 (49.8 %). e2e traffic in the boot window read 0.52–0.67
-  acceptance. Against the no-spec shape (48.0 tok/s): **+31 % at the
-  recipe temp** — the drafter earned its 2.4 GiB.
-- **Vision**: 512×512 solid-red PNG → `Red` at a 128-token budget (64
-  tokens: all thinking, empty content). `prompt_tokens: 321` confirms the
-  image part reached the model.
-- **Prompt cache**: `cached_tokens: 317` of 321 on the second probe — the
-  slot prefix cache is live.
-- Deploy: commit da36fa3, e2e gate 33/33 green on the live fleet.
-
-## Measured: the 2 × 114,688 / ubatch-1024 boot (2026-10-06 late, historical)
-
-- **argv** (from `/proc/<MainPID>/cmdline`): `--ctx-size 229376
-  --parallel 2` (per-slot 114,688), `--batch-size 4096 --ubatch-size
-  1024`, `--spec-type draft-mtp`. Deploys 87721ab (the rename — old
-  `frank-small.service` stopped + disabled first, by hand: `disable` on
-  a relinked unit removes only its symlink) then f2575a1 (the shape);
-  health gate + MainPID check green, e2e 32 passed / 1 flaky-retry.
-- **VRAM: 22,374 used / 1,751 MiB free** (nvidia-smi `free` field) —
-  the per-slot cut (≈1.05 GiB of KV) paid for the ubatch-1024 buffers
-  and netted back ~1.25 GiB against the evening's 489 MiB; back above
-  the morning's 1,397 MiB.
-- **Decode + MTP acceptance** (live `timings`): recipe (temp 1.0) →
-  **58.2 t/s** (430/1005, 42.8 %); temp 0 → **66.9 t/s** (475/873,
-  54.4 %). The 2 × 131,072 boot measured 63.3 / 72.9 — −8–10 %, but
-  acceptance is text-dependent and these prompts differ from that
-  measurement's, so the comparison is CONFOUNDED, not a shape verdict;
-  a clean A/B needs stop + mask (bench policy).
-- **Smoke**: `READY` completion + `reasoning_content` populated; the
-  harness reached :7771 under the new alias (models.yml provider/id
-  `frank-zealot`, config.yml modelRoles rewritten; per-slot
-  `contextWindow: 114688`).
-
-## Measured: ubatch 2048 (2026-10-06 midnight, live)
-
-- **argv**: `--ctx-size 229376 --parallel 2 --batch-size 4096
-  --ubatch-size 2048 --spec-type draft-mtp` (commit d8da7a9, Frank's
-  "set ubatch to 2k"; deploy green, e2e gate 33/33).
-- **VRAM: 23,120 used / 1,005 MiB free at boot** — the 1024 → 2048 rung
-  cost **746 MiB**, ~130 over the 616 MiB the rung measured on
-  2026-10-05 (rung cost is shape-dependent, not a constant). The
-  512×512 solid-red vision probe (`Red`) then took ~240 MiB **and kept
-  it**: settled **763 MiB free** — on this boot the vision encoder's
-  buffers stay resident after the request (the evening boot saw its
-  34 MiB transient return).
-- **Decode (ubatch-invariant knob, same prompt as the 1024 rung)**:
-  recipe → 52.0 t/s (388/1133, 34.2 %) vs 58.2 t/s / 42.8 % at 1024 —
-  acceptance variance at temp 1.0, not a ubatch effect.
-- **Prefill (THE ubatch knob)**: **1,209 t/s @ 3,760 tokens** — first
-  recorded number for this shape; future rungs compare against this
-  (no same-prompt 1024 baseline exists; the law predicts ≈ +4.6 %
-  1024 → 2048).
-- **Headroom left ~0.7–1.0 GiB.** ubatch 4096 (= the logical batch, no
-  micro-splitting) doubles the buffer growth again — not affordable;
-  **2048 is the ceiling for this shape.**
-
-## Client contract
-
-- Reasoning in `reasoning_content` (`--reasoning-format deepseek`);
-  thinking tokens COUNT against `max_tokens`. **This payload thinks more
-  than the censored Swift 1.5** (uncensored finetune) — short `max_tokens`
-  budgets can be eaten entirely by thinking (`finish_reason: length`,
-  empty content); the 32,768 `maxTokens` in models.yml is comfortably
-  above.
-- The embedded chat template gates `reasoning_effort` to
-  `{xhigh, medium, low}` — anything else is a raise/HTTP 500, so
-  `models.yml` `compat.reasoningEffortMap` folds the harness levels onto
-  that set (mandatory, same as every Qwen3.8-template endpoint).
-- Sampling rides the wire (`extraBody`): the publisher recipe temp 1.0 /
-  top_p 0.95 / top_k 20 / min_p 0 (evaluated at ctx 131,072).
-- The RCO-IQ3_XXS tier is a **per-tensor allocation mix** (IQ3_S 97 /
-  BF16 96 / IQ3_XXS 83 / IQ2_S 72 / IQ4_XS 33 / IQ2_XS 32 / IQ2_XXS 30 /
-  Q2_K 28 / Q4_K 19 / Q6_K 8 / IQ1_S 4 / IQ1_M 4) — all stock quant types;
-  stock llama.cpp loads it. Publisher KLD vs Swift BF16: mean 0.002631,
-  90th 0.042136, max 34.49 — RCO is the publisher's "more experimental"
-  family.
-
-## Rollback
-
-**Predecessor shape (same 2 × 114,688, ubatch 1024) is ONE revert
-away** — revert d8da7a9 and redeploy. The 2 × 131,072 / ubatch-512
-era is one revert further (f2575a1). The 2 × 131,072 no-spec
-IQ3_S-mtp shape is one revert further — revert da36fa3 next: the env
-trio then ships `SPEC_TYPE=none`, `CTX_SIZE=262144`, `PARALLEL=2`,
-`UBATCH_SIZE=512` against `/home/dank/models/Swift15-IQ3S/` (the
-IQ3_S-mtp payload, still on disk, measured 48.0 tok/s no-spec). The
-1 × 131,072 MTP shape is two reverts further back (812d9eb's parent):
-the same IQ3_S-mtp file with `SPEC_TYPE=draft-mtp`, `UBATCH_SIZE=1024`,
-`CTX_SIZE=131072`, `PARALLEL=1` (measured 58.9–66.9 tok/s with the head
-drafting).
-
-Three payloads on disk, all fetchable by the same script's git-history
-recipes:
-
-1. **Swift-1.5 IQ3_S-mtp** — the payload live until this cutover:
-   `/home/dank/models/Swift15-IQ3S/`; the IQ3_XXS swap above.
-2. **MiMo-V2.6 Q6_K 2×200,704 f16 KV** — the payload that was LIVE until
-   the 2026-10-05 cutover: `/home/dank/models/MiMo-Q6K/`; revert
-   [[frank-zealot-llama-mimo-q6k-2x200704]] (env trio at commit e5893dc or
-   380aafa's parent), one `MODEL_FILE` + shape edit.
-3. **Ornith-1.5-9B Q8_0** — the committed-but-never-deployed swap:
-   `/home/dank/models/Ornith-Q8/`; revert to 380aafa's tree (ONE native
-   262,144 slot, f16/f16 KV).
-
-## Trail
-
-- The Ornith-1.5-9B Q8_0 swap (77decb5 → 380aafa, earlier 2026-10-05) was
-  committed and pushed but its deploy never completed (blocked at the
-  whole-tree docs gate); the live service was still MiMo Q6_K when Swift
-  cut over. Both Ornith payloads were verified on disk before Swift.
-- Swift header facts (block_count 65, interval 4, KV geometry, nextn) were
-  read from the Hub file by HTTP range + struct parse — no download needed
-  — before the payload was fetched.
-- Cutover 2026-10-05 ~23:26: the radiance-cutover session (parallel)
-  label-committed both sets at Frank's direction — 1919ad7 (frank-archon
-  radiance) then baf33d0 (this swap: trio, seam, runbooks, fleet docs).
-  Deploy ran `deploy.sh dankclaw --service frank-zealot`; the service
-  dispatch and health gate went green; the first e2e run failed ONE spec
-  (12-crawl, `net::ERR_ABORTED` on `www.franksdank.cloud` — a transient,
-  unrelated to frank-zealot), green on rerun.
-- The unit's `Description=` (MiMo era) was stale at cutover time and was
-  updated in-repo afterward — the relinked unit picks it up at the next
-  deploy/restart, not retroactively.
-- 2026-10-06 (Frank): drop MTP, skip ngram, ubatch 512, TWO 128k contexts
-  (812d9eb). The tight fit landed: 1,397 MiB free, vision `Red`, decode
-  48.0 t/s no-spec. Model of the trade: the drafter's ~0.4 GiB + ubatch
-  ~0.3 GiB bought the second 4,352 MiB KV slot.
-- 2026-10-06 (Frank, evening): swap to the UNCELERSED Swift-1.5 at
-  RCO-IQ3_XXS, MTP back on (da36fa3). The lighter quant (−1.56 GiB) funds
-  the draft ctx (~2.4 GiB measured — the drafter's KV scales with the
-  target's full 262,144 context, not the 3-token proposal depth) plus the
-  second 131,072 slot: 489 MiB free, vision `Red` at a 128-token budget,
-  decode 63.3 t/s recipe / 72.9 temp 0 (48.0 no-spec).
-- 2026-10-06 (late, Frank): the endpoint renamed frank-small →
-  **frank-secondary** (87721ab — "rename frank-small → frank-secondary", scripts, unit, seam, runbooks,
-  identifiers; historical references to the OLD FreeToken/CUDA
-  frank-small, and to the Dankserver service that became frank-medium,
-  stay frank-small) and the per-slot window cut 131,072 → 114,688 to
-  fund ubatch 512 → 1024 (f2575a1). Live unit transition done by hand
-  before the deploy: `systemctl stop frank-small` + `systemctl disable
-  frank-small`, then `deploy.sh dankclaw --service frank-zealot`.
-  1,751 MiB free; decode 58.2 t/s recipe / 66.9 temp 0; models.yml +
-  config.yml retargeted.
-- 2026-10-06 (midnight, Frank): "set ubatch to 2k" → `UBATCH_SIZE=2048`
-  (d8da7a9). Cost 746 MiB: 23,120 used / 1,005 free at boot, settling
-  **763 MiB free after a vision probe** (encoder buffers stay resident
-  on this boot). Decode 52.0 t/s recipe on the shared prompt (58.2 at
-  the 1024 rung — acceptance noise, decode is ubatch-invariant);
-  prefill 1,209 t/s @ 3,760 tokens (first recorded baseline). 2048 is
-  the ceiling for this shape; 4096 would not fit.

serverAppConfigs-docs/LLM/frank-zealot-qwen38-gsq-rco.md

@@ -1,22 +0,0 @@
----
-type: reference
-title: "frank-zealot on Dankclaw — llama.cpp, Qwen3.8-27B GSQ-RCO IQ3_S"
-description: "The RETIRED frank-zealot runbook (2026-09-16 → 09-17) — mainline llama.cpp on Dankclaw's RTX 3090 serving ISTA-DASLab Qwen3.8-27B-GSQ-RCO at IQ3_S with the model's own in-file MTP head and the BF16 vision projector: identity, KV arithmetic, deployed flags, the MTP contract, the client contract, and what to check when it misbehaves."
-tags:
-  - llm
-  - frank-zealot
-  - dankclaw
-  - llama.cpp
-  - rtx3090
-  - qwen35
-  - mtp
-  - vision
-timestamp: 2026-09-16
----
-
-**RETIRED 2026-09-16 → 2026-09-17** (AGENTS.md Runbook Index); superseded by
-[[frank-zealot-gemma4-12b]], and :7771 today runs
-[[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML fork build 10754,
-`Ternary-Bonsai-2-27B-PQ2_0.gguf`). The payload is **not** a rollback target any
-more: the 2026-10-18 census of `/home/dank/models/` has no GSQ-RCO directory, so
-restoring this shape means re-downloading from ISTA-DASLab.

serverAppConfigs-docs/LLM/frank-zealot-sglang-mimo-v26.md

@@ -1,187 +0,0 @@
----
-type: reference
-title: "frank-zealot on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (RETIRED)"
-description: "The RETIRED frank-zealot runbook (2026-10-01, hours) — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode ghost, the CUDA 13 nvcc toolchain the JIT needs, and the measured numbers kept for the record. Replaced by [[frank-zealot-llama-mimo-q6k]] the same night."
-tags:
-  - llm
-  - frank-zealot
-  - dankclaw
-  - sglang
-  - rtx3090
-  - qwen
-  - mtp
-  - vision
-  - int8
-timestamp: 2026-10-01
----
-
-# frank-zealot — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP)
-
-> **RETIRED 2026-10-01, hours after cutover** (AGENTS.md Runbook Index). Replaced by
-> [[frank-zealot-llama-mimo-q6k]] — llama.cpp CUDA on the MiMo Q6_K GGUF,
-> chosen for the asymmetric KV (f16 K + q8_0 V) SGLang cannot express. :7771
-> today runs [[frank-zealot-llama-prism-bonsai9-mtp]] (llama.cpp PrismML fork
-> build **10777**, `Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, 2 × 262,144,
-> exact f16 K + f16 V, MTP off by measurement).
-> The venv (`/home/dank/GitPackages/sglang-cu128/`) is still on disk; the INT8
-> payload was staged to
-> `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 and
-> **that stage batch is gone** (`ls /home/dank/.AGENTTRASH/`, 2026-10-18 —
-> only the 20261005/06/07/10/17 batches remain), so this runtime needs
-> `old/download-frank-zealot-sglang-int8.sh` to re-stage.
-
-## Identity
-
-- **Endpoint:** `frank-small` at `192.168.0.143:7771` (Dankclaw) — the id
-  became `frank-secondary` on 2026-10-06 and `frank-zealot` on 2026-10-09.
-- **Runtime:** SGLang (latest) in a self-contained python3.12 venv at
-  `/home/dank/GitPackages/sglang-cu128/` (torch 2.13.0+cu130,
-  transformers 5.12.1, flashinfer 0.6.18, sgl-kernel). No container — a root
-  systemd unit `scripts/frank-zealot.service` (`User=dank`,
-  `LimitMEMLOCK=infinity`) launched by `scripts/serve-frank-zealot.sh` with
-  tunables from `scripts/frank-zealot.env`.
-- **Model:** `letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`
-  (9B dense, 12.0 GiB, INT8 W8A16 AutoRound group-128 symmetric).
-
-## Shape
-
-- **Architecture `qwen3_5` hybrid:** 32 layers = 8 full-attention
-  (4 KV heads x 256 head_dim, `partial_rotary_factor: 0.25`) + 24 Gated
-  DeltaNet. `max_position_embeddings: 262144`, and a `qwen3_5_vision`
-  tower (27 layers) in the safetensors — no separate mmproj file.
-- **Context:** two 262,144-token slots at the model's native window
-  (`--context-length 262144`, `--max-running-requests 2`) against a
-  **pinned** 606,208-token pool (`--max-total-tokens`) — 2026-10-01
-  reslice from four 151,552-token slots: 2 x 262,144 = 524,288 leaves
-  ~82k tokens of pool slack, and a compaction request (frank-archon's
-  trigger 183,501 + a 32,768-token summary budget = 216,269) now fits
-  one request. A single request may grow to the whole pool — a
-  170,019-token prompt was verified at the 4-slot shape — the grid is
-  the two-way concurrency contract. **That compaction arithmetic is
-  specific to a 262,144-token slot**: the shape that followed on 2026-10-02
-  ran three slots of 180,224, which 183,501 tokens already exceeds — see *What
-  three slots cost* in [[frank-zealot-llama-mimo-q5km]]; :7771 today runs two
-  262,144-token conversations over one 524,288-position pool with exact f16 KV.
-- **KV:** `fp8_e4m3` — 16 KiB/token (8 attn layers x 2 x 4 heads x 256
-  dim x 1 B), pool 9.31 GiB.
-- **The memory shape (the 4.5 GiB the naive math misses):** the DeltaNet
-  ssm state pool is a first-class consumer. The engine autofills the
-  mamba slot count to spend a fixed ~5.1 GB budget *regardless of dtype*
-  (fp32 at 110 slots, bf16 at 217 — same 5.1 GB), so both dials are
-  pinned: `--mamba-ssm-dtype bfloat16 --max-mamba-cache-size 32` = 795
-  MiB. With that, weights 11.35 GiB + mamba 0.78 GiB + pool 9.31 GiB
-  leave ~0.8 GB — enough for eager-prefill workspaces, NOT enough for
-  CUDA-graph capture (autofit's 687,388-token pool leaves 0.53 GB and
-  dies in capture, or later in the first long request once Triton's
-  deferred kernel loads eat the remainder). Hence
-  `--disable-prefill-cuda-graph`; decode graphs are untouched.
-- **Quantization:** `--quantization auto-round` — the checkpoint's
-  `quantization_config.json` declares `quant_method: "auto-round"` and
-  SGLang loads it natively (vision tower, `embed_tokens` and
-  `linear_attn.in_proj` stay fp16).
-
-## Spec method
-
-**None — and this is a checkpoint fact, not a tuning choice.** The
-`text_config` declares `mtp_num_hidden_layers: 1`, but the shipped
-safetensors carry **zero `mtp.*` tensors**: verified 2026-10-01 against
-`model.safetensors.index.json` for both this INT8 conversion and the base
-`XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` (32 layers, indices 0–31, no
-`mtp.`/`nextn.`/`layers.32` keys in either). With NEXTN enabled, SGLang
-boots a 4.24 GiB draft model (`Qwen3_5ForCausalLMMTP`) from nothing and
-decodes at `accept len: 1.00, accept rate: 0.00` — every proposal rejected,
-paying the 4-token verify tax for no token. The serve script passes no
-`--speculative-*` flag, and the seam gates that.
-
-## CUDA 13 toolchain (the sharp edge)
-
-The venv's torch is cu130, so every JIT kernel (sgl-kernel marlin repack,
-flashinfer) compiles with a **CUDA 13.0 nvcc extracted from debs into the
-venv** at `cuda-nvcc/usr/local/cuda-13.0/` — the system nvcc is 12.4 (no
-`--compress-mode`) and its headers clash with the 13.0 device passes.
-
-Components (all from the ubuntu2404 CUDA repo, `dpkg -x`'d, not installed):
-
-| Package | Supplies |
-|---|---|
-| `cuda-nvcc-13-0` (13.0.88) | `bin/nvcc`, `ptxas`, `cudafe++`, … |
-| `libnvvm-13-0` (13.0.48) | `nvvm/bin/cicc` (the device compiler) |
-| `cuda-cudart-dev-13-0` (13.0.88) | `include/cuda_runtime.h`, `lib64/stubs` |
-| `cuda-crt-13-0` (13.0.88) | `include/crt/host_runtime.h` (2-arg `__cudaLaunch`) |
-| `libnvptxcompiler-13-0` (13.0.88) | static lib + `nvPTXCompiler.h` |
-| `cuda-cccl-13-0` (13.0.50) | `cccl/cuda/...` headers (libcudacxx) |
-
-Three local modifications the tree carries:
-
-1. **`bin/nvcc` is a shim** over `bin/nvcc.real` adding
-   `-allow-unsupported-compiler` (host GCC 16 > nvcc's max 13).
-2. **`targets/x86_64-linux/include` and `lib` are symlinks** to
-   `../../../include` / `../../../lib64` (the deb ships them as real
-   near-empty dirs; the stub pass must see the 13.0 headers, not
-   `/usr/include` 12.4).
-3. **`include/cuda` → `cccl/cuda`** symlink (CCCL 3.x layout; flashinfer
-   includes `<cuda/functional>`), and **`crt/math_functions.h`** patched with
-   `noexcept` on `rsqrt`/`rsqrtf` (glibc 2.42 declares them `noexcept(true)`;
-   the unpatched 13.0 declarations are incompatible in the C++ host stub).
-
-JIT caches: `~/.cache/sglang/jit/sm86/` and
-`~/.cache/sglang/.cache/flashinfer/0.6.18/86/` — clear both when a toolchain
-change lands.
-
-## Payload
-
-Eight safetensors shards (12,891,718,440 B total) + tokenizer and config
-files lived under `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound/`
-(absent from the 2026-10-18 census — see the banner),
-all LFS sha256-verified by `old/download-frank-zealot-sglang-int8.sh`
-(`--verify-only` re-checks) — the script moved to `old/` when the endpoint
-cut over to llama.cpp + GGUF on 2026-10-01 night, and
-`scripts/download-frank-zealot.sh` now fetches the GGUF quant instead. The
-serve script refuses to start with a missing shard.
-
-## Client contract
-
-- **Alias:** `frank-small` (the `id` at `/v1/models`).
-- **Reasoning:** extracted to **`reasoning_content`** (live-verified
-  2026-10-01 — this SGLang build's qwen3 parser uses the same field name
-  as llama.cpp; models.yml sets `reasoningContentField:
-  "reasoning_content"`). `--reasoning-parser qwen3`.
-- **Tools:** `--tool-call-parser qwen3_coder`.
-- **Vision:** supported (in-checkpoint tower).
-- **Sampling:** SGLang has NO server-wide `--temperature`/`--top-p`/
-  `--top-k` — sampling rides on the request; the harness sends the Qwen
-  recipe (temp 0.7, top_p 0.8) via models.yml `compat.extraBody`.
-- **Reasoning effort:** the full OpenAI set is accepted (the chat template
-  has no effort gate).
-
-## Measured
-
-- Boot ~60 s with warm JIT caches (several minutes cold — the first boot
-  compiles the sgl-kernel + flashinfer kernels; the CUDA 13 toolchain
-  above is what makes that compile succeed).
-- Decode **83.9 tok/s** (no spec decode — nothing to accept).
-- Prefill ~2,050–3,370 t/s eager (4-slot shape: a 170,019-token prompt in
-  50.5 s, ~3,370 t/s; 2-slot shape: a 200,000-token prompt in 97.3 s wall
-  incl. 8 decode tokens, ~2,056 t/s).
-- 4-slot shape: 4 concurrent 54,027-token requests all answered in ~17 s
-  (216k tokens of live KV at once). 2-slot shape: two concurrent
-  50,000-token requests both answered (28.8 s wall).
-- **Compaction fit (the reslice's purpose):** a 200,000-token prompt —
-  well past frank-archon's 183,501-token compaction trigger (70% of its
-  262,144 window) — prefills and completes in ONE request.
-- Vision: a 64x64 red square answers "Red" with `image_tokens: 64`.
-
-## Rollback
-
-[[frank-zealot-llama-mimo-q5km]] was the live shape two steps forward; the
-practical rollback at the time was [[frank-zealot-llama-mimo-q6k]] (payload at
-`/home/dank/models/MiMo-Q6K/`, still on disk 2026-10-18).
-This era's own rollback, the llama.cpp Gemma4-12B QAT payload, was staged to
-`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/` on
-2026-10-02; its unit/env/serve trio remains in `old/`.
-
-# Citations
-
-- Model: https://huggingface.co/letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound
-- SGLang: https://github.com/sgl-project/sglang
-- Serve script: `scripts/serve-frank-zealot.sh`; tunables: `scripts/frank-zealot.env`

serverAppConfigs-docs/LLM/frank-zealot-vllm-gemma4.md

@@ -1,226 +0,0 @@
----
-type: reference
-title: frank-zealot on Dankclaw — vLLM, Gemma4-12B QAT w4a16 (compressed-tensors)
-description: "The RETIRED frank-zealot runbook (2026-09-19, same-day revert; rollback target) — vLLM on Dankclaw's RTX 3090 serving google/gemma-4-12B-it-qat-w4a16-ct (Marlin int4 weights, bf16 KV, 262k window) with Google's bf16 MTP assistant, the audio-capable derived image, the no-nvidia-container-toolkit device mounts, measured throughput, and the drafter-Marlin bug that killed the w4a16 assistant. Replaced by [[frank-zealot-gemma4-12b]] on the same day."
-tags:
-  - llm
-  - frank-zealot
-  - dankclaw
-  - vllm
-  - rtx3090
-  - gemma
-  - mtp
-  - vision
-  - audio
-timestamp: 2026-09-19
----
-
-# frank-zealot — vLLM on the RTX 3090 (Gemma4-12B QAT w4a16)
-
-> **RETIRED 2026-09-19 (same-day revert) — rollback target only in git.**
-> This vLLM deployment was cut
-> back to [[frank-zealot-gemma4-12b]] (llama.cpp — itself retired
-> 2026-10-01; :7771 today runs [[frank-zealot-llama-prism-bonsai2-pq20]])
-> the same day it landed. Same port
-> (7771), same alias (`frank-small` — the name `frank-zealot` dates from
-> 2026-10-06/10-09), same client contract surface — the
-> runtime, the model payload, the drafter and the service shape (root
-> systemd unit → rootless Quadlet) all changed.
-
-## Identity
-
-| what | value |
-|---|---|
-| runtime | vLLM `vllm/vllm-openai` dev build `gemma-x86_64-cu129` (base `sha256:e24d3d8e…001e`), audio extra pip-installed into `localhost/vllm/vllm-openai:frank-small` (`8dad46d79c46` — still in rootless storage on the 2026-10-18 `podman images` census; the tag carries the era name) |
-| unit | **rootless Quadlet** `dankclaw/frank-small/quadlet/frank-small.container` (era path; deleted by the revert commit `374041c` — root systemd unit retired 2026-09-19) |
-| model | `google/gemma-4-12B-it-qat-w4a16-ct` — compressed-tensors w4a16 (int4 Marlin weights, 10,264,229,896 B) |
-| model dir | `/home/dank/models/gemma4-12b-qat-w4a16-ct/` (absent from the 2026-10-18 census of `/home/dank/models/` — re-fetch to use this rollback) |
-| drafter | `google/gemma-4-12B-it-assistant` **bf16** (845,719,296 B) at `/home/dank/models/gemma4-12b-assistant-bf16/` (absent from the same census) |
-| port | 7771 → 8000 (alias `frank-small` in this era; `frank-dragoon` keeps 7770) |
-| window | 262,144; KV pool **289,647 tokens** bf16; **2 concurrent full-context sessions** |
-| multimodal | native: text + image + **audio** (derived image carries `av`/`scipy`/`soundfile`/`mistral_common[audio]`) |
-
-## Why vLLM
-
-- The w4a16-ct checkpoint is a compressed-tensors format; vLLM's Marlin
-  kernel is its native int4 GEMM path. llama.cpp could not load it at all.
-- Native audio encoder in the checkpoint — the fleet's only audio-capable
-  endpoint moves here (llama.cpp's path needed the mmproj audio tower).
-- CUDA graphs (PIECEWISE 21/21 + FULL 11/11) — the llama.cpp CUDA build on
-  this host had no graph support.
-
-## The drafter-Marlin bug (why the assistant is bf16)
-
-The w4a16-ct assistant (`kunhunjon/gemma4-12b-qat-assistant-w4a16-ct`)
-**cannot draft in vLLM**: the drafter's linear layers are loaded through a
-path that does not apply the Marlin kernel, so the packed int4 weights are
-misread as bf16 and every draft is garbage — measured acceptance **1.00**
-(pure pass-through). There is no "Using MarlinLinearKernel" log line after
-the drafter loads. Google's **bf16** assistant (`google/gemma-4-12B-it-assistant`)
-has no quantization to miss, and measures **acceptance 2.80–3.56**
-(per-position 0.77 / 0.57 / 0.52 / 0.42, ~57 %). The 211 MB delta
-(634 → 845 MB) is absorbed by pool rounding: both assistants yield the
-identical 289,647-token KV pool.
-
-## KV arithmetic — why two full-context sessions
-
-Gemma4 hybrid: 42 SWA layers (window 1024) + 6 full-attention layers, MHA
-16 heads × 512 dim. The SWA layers cache only the 1024-token window, so
-per-token KV is dominated by the full-attention layers:
-6 × 16 × 512 × 2 (K,V) × 2 B = **196.6 KiB/token** (bf16).
-289,647 tokens × 196.6 KiB ≈ 55 GiB of addressable pool… which does not
-sound like it fits a 24 GiB card — it fits because vLLM's SWA pool
-shares the 1024-window ring across all concurrent requests, so the
-marginal cost of a *second* 213k session is ~6 full-attention GiB, not
-a second copy of everything.
-
-Measured (salted, 2026-09-19):
-
-| load | KV usage | result |
-|---|---|---|
-| 1 × 213k | ~35 % | runs |
-| 2 × 213k | **69.3 %** | both run in parallel (17.5 % slowdown each) |
-| 4 × 213k | — | `Running: 2, Waiting: 2` — the 3rd queues |
-
-**Ceiling: 2 concurrent full-context sessions.** (The llama.cpp
-deployment fit 4 slots of 262k with q8 KV; the bf16 KV here is 2× the
-bytes, and vLLM's pool arithmetic is coarser. This is the accepted
-trade for the native w4a16 format + audio + CUDA graphs.)
-
-## Measured throughput (salted, 2026-09-19)
-
-| probe | tokens | wall | rate |
-|---|---|---|---|
-| prefill @ 12.7k | 12,706 | 27.2 s | 466 t/s (degraded — concurrent traffic) |
-| prefill @ 67k | 67,252 | 52.0 s | **1,293 t/s** |
-| prefill @ 213k | 213,044 | 350.3 s | **608 t/s** |
-| 2 × 213k prefill | 213,044 each | 374 s batch | 608 → 570 t/s per stream |
-| decode @ 213k depth | 300 out | 372 s total | **~13.6 t/s** (bandwidth-bound) |
-| decode @ 85k depth | 300 out | 82.3 s total | **~17.6 t/s** |
-
-- **Prefill degrades with depth** (1,293 → 608 t/s): the 6 full-attention
-  layers re-read the growing KV, and TRITON_ATTN is not as depth-efficient
-  as llama.cpp's flash-attn path here.
-- **Decode at 213k is memory-bandwidth-bound**: ~8.4 GB of KV read per
-  token against the 3090's ~936 GB/s gives a ~10–14 t/s ceiling; 13.6 is
-  right at it. At normal depths decode is far faster (the 67k-depth probe
-  is in the bench log). MTP acceptance ~3.2–3.6 is folded into these
-  numbers.
-- **Bench gotcha (cost an hour)**: a repeated-paragraph filler does NOT
-  scale linearly in tokens — BPE compresses the repetition harder as the
-  span grows (1.4M chars ≈ 262k tokens, not the ~175k a linear fit
-  predicts). Size fillers with the model's own tokenizer
-  (`AutoTokenizer.encode` on the exact prefix), not char counts.
-- `POST /flush_cache` is **404** in this build — salt every probe
-  (uuid prefix) instead.
-
-## The image
-
-`dankclaw/frank-small/Dockerfile` (era path, deleted by the revert commit
-`374041c`; recover with `git show 374041c^:dankclaw/frank-small/Dockerfile`) — 16 comment lines + `FROM
-localhost/vllm/vllm-openai:gemma4-unified` + `RUN pip install
---no-cache-dir av scipy soundfile "mistral_common[audio]"`. **No COPY
-lines** — the earlier `triton-sm89-sitecustomize.py` hack is gone:
-`sm_86` (Ampere GA102) is the *correct* compute capability, confirmed
-after the driver swap, so no arch override is needed.
-
-## No nvidia-container-toolkit — manual CUDA mounts
-
-Dankclaw has no nvidia-container-toolkit, so the Quadlet unit wires CUDA
-by hand:
-
-- `Device=` for `/dev/nvidia0`, `/dev/nvidiactl`, `/dev/nvidia-uvm`,
-  `/dev/nvidia-uvm-tools`
-- host `libcuda.so.1`, `libcuda.so`, `libnvidia-ml.so.1` mounted read-only
-  into **both** `/usr/lib/x86_64-linux-gnu/` and `/usr/local/nvidia/lib64/`
-- `NVIDIA_VISIBLE_DEVICES=all`, `--ulimit memlock=-1:-1`
-
-If the driver is upgraded on the host, these mount sources move with it
-(they are versionless symlinks) — no unit change needed.
-
-## Host driver note
-
-Dankclaw is **Ubuntu 26.04 LTS** (not CachyOS — that is the dev box).
-Driver `nvidia-driver-610` **610.57.04** (installed 2026-09-19, replaces
-595.91.07). `nvidia-smi` reports compute capability **8.6** — correct for
-the GA102/Ampere 3090; the earlier sm_89 confusion was a driver artifact,
-not a hardware mystery.
-
-## Deployed config
-
-Repo truth **for this era** (the files carry llama.cpp knobs today): `scripts/frank-zealot.env` + `scripts/serve-frank-zealot.sh`
-(mounted into the container at `/scripts/`, which is the entrypoint) +
-`dankclaw/frank-small/quadlet/frank-small.container` (era path, deleted at `374041c`). Key values:
-`KV_CACHE_DTYPE=auto` (resolves to bf16 — fp8_e5m2 is refused by
-compressed-tensors checkpoints and fp8 is hardware-dead on sm_86),
-`ENFORCE_EAGER=0` (CUDA graphs on), `max_model_len=262144`, MTP via
-`--speculative-config` pointing at the bf16 assistant. `TRITON_OVERRIDE_ARCH`
-is **removed**. The serve script refuses to start if the weights, the
-assistant, or the vllm binary are missing.
-
-## Client contract
-
-- **Reasoning lives in `message.reasoning`** (vLLM's gemma4 reasoning
-  parser), not `reasoning_content` — the harness maps it via
-  `compat.reasoningContentField: "reasoning"` in `~/.omp/agent/models.yml`.
-  The field is empty when the model emits no thinking tag (behavior, not
-  a parser bug).
-- `preserve_thinking: true`, `temperature`, `top_p` in `extraBody` are
-  accepted (200, not 400) — vLLM ignores the unknown key.
-- Every `reasoning_effort` level accepted (all seven probed 2026-09-17).
-- Tools → `tool_calls` (verified `get_weather`).
-- Image → 200, described correctly (native vision tower).
-- Audio → 200, a 440 Hz WAV tone described correctly (native audio
-  encoder; verified in the derived image).
-- Harness: `contextWindow: 262144`, `maxTokens: 32768`,
-  `input: [text, image, audio]` — era values; the endpoint reads 200,704 /
-  16,384 / `[text, image]` today (`~/.omp/agent/models.yml:206,212,214`), and
-  audio left the fleet with this payload on 2026-10-01.
-
-## Boot ledger
-
-t=135 s to ready (weights are 10.3 GB int4 + 0.85 GB assistant from a
-bind mount; Marlin active; CUDA graphs PIECEWISE 21/21 + FULL 11/11;
-KV pool 289,647 tokens). Health is 503 until ready — do not kill the
-container while it loads.
-
-## Troubleshooting
-
-- **Acceptance 1.00** — the drafter is the w4a16-ct assistant again
-  (drafter-Marlin bug). Confirm the assistant mount points at
-  `gemma4-12b-assistant-bf16`.
-- **`sm_89` in an error** — stale arch override; `TRITON_OVERRIDE_ARCH`
-  must be absent. 8.6 is correct.
-- **Audio 400/500** — the container is running the *base* image, not the
-  derived `:frank-small` (missing `av`). In this era, check the deployed image
-  id is `8dad46d7…` (`podman images`, 2026-10-18: it still exists, unused —
-  nothing on :7771 runs in podman today).
-- **CUDA not visible in container** — a `Device=` line or a lib mount
-  dropped out of the unit after an edit; the unit is the source of truth,
-  the host-side generated unit is not.
-- Logs: `journalctl --user -u frank-zealot` (rootless Quadlet in this era;
-  `podman logs` covers only the current `--replace --rm` instance). The unit is
-  a ROOT unit today: `journalctl -u frank-zealot`.
-
-## Rollback
-
-[[frank-zealot-gemma4-12b]] — the llama.cpp deployment. Its weights,
-mmproj and MTP sidecar are **no longer on the host** under
-`/home/dank/models/Gemma4-12B-QAT/` (2026-10-18 census: the directory is
-gone — it was staged to `.AGENTTRASH/20261002-retired-llm-payloads/` and that
-stage dir is gone too), so this rollback re-fetches
-`HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` and restores the era
-trio from git (`git show 374041c:scripts/frank-small.service`,
-`…:scripts/frank-small.env`, `…:scripts/serve-frank-small.sh`); the era quadlet
-lived at `dankclaw/frank-small/quadlet/frank-small.container`. One owner per
-port: the llama.cpp unit is root, so stop the container unit before starting it.
-
-## References
-
-- `scripts/serve-frank-zealot.sh` — the actual exec line (today: llama.cpp; the
-  vLLM exec line is in `git show 374041c^:scripts/serve-frank-small.sh`)
-- `dankclaw/frank-small/Dockerfile` — the audio-derived image (era path, deleted at `374041c`)
-- `dankclaw/frank-small/quadlet/frank-small.container` — the unit (era path, deleted at `374041c`)
-- [[frank-zealot-gemma4-12b]] — RETIRED predecessor (rollback)
-- [[halogen-flash-server]] — the sibling endpoint on the iGPU
-- [[VLLM Overview]] / [[VLLM Index]] — runtime background

serverAppConfigs-docs/LLM/frank-zealot.md

@@ -0,0 +1,191 @@
+---
+type: reference
+title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10777), Ternary-Bonsai-2-9B PQ2_0-MTP, 2 × 262144, exact f16 KV"
+description: "The one frank-zealot reference: the live shape on Dankclaw's RTX 3090 (:7771) exactly as scripts/frank-zealot.env ships it, every era with its rollback path, the deduplicated laws of this card, the wire contract, and the owning files."
+tags:
+  - llm
+  - frank-zealot
+  - dankclaw
+  - llama-cpp
+  - gguf
+  - rtx3090
+timestamp: 2026-10-18
+---
+
+# frank-zealot — llama.cpp (PrismML) Bonsai-2-9B PQ2_0-MTP, 2 × 262,144, exact f16 KV
+
+Dankclaw's RTX 3090 endpoint — `http://192.168.0.143:7771`, wire id `frank-zealot`, the fleet's **compaction hub**
+(widest window it has). One page per endpoint: eras are rows below, not pages. Siblings: [[frank-archon]],
+[[frank-dragoon]]. Deploy with `./scripts/deploy.sh dankclaw --service frank-zealot`; swap mechanics in
+[[Model Swap and Rollback]], sizing math in [[Llama Cpp Serving Sizing]]. `scripts/frank-zealot.env` and
+`scripts/serve-frank-zealot.sh` are the authority for the shape; where any prose disagrees with them, the prose
+is the bug.
+
+## Current shape
+
+| Item | Value |
+|---|---|
+| Checkpoint | `/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf` — 2,675,410,720 B, LFS sha256 `c79fcdfaf6b7bf47…` |
+| Projector | `…/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf` — 624,229,664 B, sha256 `e0cb6f6988cea323…` — **Q8_0 is mandatory**, not a preference (see Contract) |
+| Fetch | `scripts/download-frank-zealot-bonsai9.sh [DEST]` (`--verify-only`; `-f`, byte count, LFS sha256 per file) |
+| Runtime | `/home/dank/GitPackages/llama-prism-b10777/llama-server` — PrismML **fork**, `0.2.0-dev (build 10777, commit e3de71522)`; the only loader for this payload |
+| Trio | `scripts/frank-zealot.service` (root, relinked, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`, `TimeoutStopSec=180`) · `scripts/frank-zealot.env` · `scripts/serve-frank-zealot.sh` |
+| Address / alias | `BIND_ADDR=0.0.0.0`, `LISTEN_PORT=7771`, `ALIAS=frank-zealot` (a flag, not a fact) |
+| Flags | `--ctx-size 524288 --parallel 2` ⇒ 262,144 per slot · `--cache-type-k f16 --cache-type-v f16` (exact, **16,384 MiB**) · `--flash-attn on` · `--batch-size 4096 --ubatch-size 2048` · `--n-gpu-layers 99` · `--jinja --reasoning auto --reasoning-format deepseek` · `--metrics` · `--mmproj` · `--log-verbosity 4` · `--presence-penalty 1.0` · **`SPEC_TYPE=none`** |
+
+**Why this shape.** `general.architecture qwen35`: 33 blocks (32 trunk + `blk.32` = the MTP block),
+`full_attention_interval 4` ⇒ 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, `nextn_predict_layers 1`
+(the head **is in the file**), trained window 262,144, `general.sampling.*` 1.0/0.95/20. Only the 8 full layers bill
+per token (32,768 B at exact f16 ⇒ 16,384 MiB over the pool) and the DeltaNet layers add ~50 MiB of recurrent state
+**per slot**: exact KV costs the third conversation and nothing else.
+
+**MTP is OFF by measurement, not by absence of a head.** The drafter is *good* — 2026-10-18 measured
+acceptance 0.798 / 0.617 per position, mean accepted length 2.41, and the publisher's card says depth 2 works
+best — and it still loses here, because the draft context is built against the *target's* pool: `blk.32` bills
+4,096 B/position at its default f16 ⇒ 2,048 MiB at 524,288 positions on a trunk that projects 21,234 MiB of the
+24,123 MiB addressable (that projection predicts 2,889 MiB free; replace it with the measured pair). Exact KV and a
+drafter cannot share this card. `--spec-draft-type-k`/`--spec-draft-type-v` (default f16, verified in this build's
+`--help`) would cut the draft to 1,088 MiB — legal on b10777, **never booted beside exact KV**. There is no
+`--spec-draft-model`: the MTP draft rides the main model's weights. The env carries the recipe
+(`SPEC_DRAFT_N_MAX=2`, `SPEC_DRAFT_N_MIN=0`, `SPEC_DRAFT_P_MIN=0`); `SPEC_TYPE=none` never sends it.
+
+**Harness contract** (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw-3090)`, id `frank-zealot`):
+`contextWindow: 262144` — **the per-slot window**, because llama.cpp divides `--ctx-size` across slots and answers a
+longer prompt with a 400, never a truncation — plus `maxTokens: 16384`, `input: [text, image]`,
+`compat.reasoningContentField: reasoning_content`, `compat.reasoningEffortMap` (**load-bearing**: the template takes
+only `xhigh|medium|low` and raises on anything else), `extraBody` 1.0 / 0.95 / 20 / `min_p 0`, and the
+`compactionModel` of [[frank-archon]], [[frank-dragoon]] **and** frank-zealot all pointing here.
+
+**The exact-KV boot, measured** (2026-10-18 evening, under the unit, verbosity 4). KV **16,384.00 MiB** =
+32,768 B/token × 524,288 positions, to the MiB — rate derivation and ledger agree with no slop, which is what
+buries the doc's old "f16 needs 26,112 MiB and cannot boot" figure for good: that was the 27B's 34,816 B/token
+applied to the 9B. RS **100.50 MiB** (2 cells, one 50.25 MiB cell lighter than the q8_0 era), compute
+1,408.06 + 1,088.07 CUDA_Host, mmproj worst case **868.32 MiB** charged *after* the engine's own fit check,
+settled `nvidia-smi` **21,176 used / 2,949 free** of 24,123, `n_slots = 2`, `n_ctx_slot = 262144`,
+`DropInPaths` empty, no clamp line, no `cudaMalloc` error — within 24 MiB of the pre-flight projection. That is
+a desktop margin: the ~2 GiB of slack is what an image storm has to live inside.
+
+**Rates on this shape are FLOORS, not the shape's rates** — measured the same evening greedy with `ignore_eos`
+at 512 tokens, first request discarded, while another consumer drove the other slot (56 new prompts in three
+minutes, 96 % GPU-util): the exact condition the bench window exists to exclude, and one contended deep arm
+even came back truncated (a 512 KB body accepted as `prompt_n = 4`).
+
+| Arm | q8_0 3-slot record | f16 2-slot, contended |
+|---|---|---|
+| single-stream decode | 168.7 | **134.1 – 141.5** (145.8 and 163.2 in a lighter minute) |
+| concurrent aggregate | ≈300 across 3 (102.8 each) | **256.6 / 262.4 / 265.1** across 2 (≈128–133 each) |
+| prefill at ~106k | 2,877 at 106,418 | **3,159.5 at 106,282** |
+| decode at 106k depth | never measured | **71.5** |
+
+What the floors already support: per-conversation concurrency **improved** (two streams hold ~130 t/s each where
+three held 102.8), endpoint aggregate sits ~12 % under the 3-slot record, prefill beat the record while
+contended, and single-stream decode is between parity and −20 %. The 1.882× KV byte bill is real; its true size
+on this card is still unmeasured, so quoting 134 or 163 t/s as "the" number is a guess in either direction —
+re-run the same arms with the box quiet before a figure is carried anywhere else.
+
+**Fallback rungs**, cheapest first — one line of `frank-zealot.env` each, and each must keep the seam's fit gate green:
+
+| Knob | Effect |
+|---|---|
+| `UBATCH_SIZE=1024` | hands back 1,306 MiB of compute buffers; prefill −17 %; it is also the arm that ran the drafter at three windows (213.5 t/s) |
+| `CACHE_TYPE_V=q8_0` | exact K + quantized V: KV 16,384 → **12,544 MiB**, freeing 3,840 (llama.cpp has no true fp8 KV dtype — q8_0 is the 8-bit class) |
+| `CTX_SIZE=401408` | two 200,704 conversations, exact KV at 12,544 MiB (same 3,840 freed) — `contextWindow` MUST follow the per-slot number |
+| `CTX_SIZE=786432 PARALLEL=3 CACHE_TYPE_K/V=q8_0` | the era below this one: 13,056 MiB of cache, the third conversation back |
+| `CACHE_TYPE_K/V=q4_0` | 10,240 B/token ⇒ 5,120 MiB (the 27B era measured q4_0 at 16,392 used / 7,733 free at 2 × 200,704) |
+
+## Eras
+
+| Window | Payload | Runtime | Shape | Replaced because | Rollback |
+|---|---|---|---|---|---|
+| 2026-10-18 13:19 → same evening (the boot `68afc6d` replaced) | the same 9B MTP file and projector | PrismML fork 10777 | 3 × 262,144 (pool 786,432), q8_0 K+V = 13,056 MiB, settled 17,956 / 6,169 MiB; 168.7 t/s decode (three runs 168.4–168.7), ~300 t/s over three streams (102.8 each), 2,877 t/s prefill at 106,418 | Frank ordered unquantized KV, and that pool cannot survive it: 24,576 MiB of f16 cache > 24,123 addressable | one env line — `CTX_SIZE=786432 PARALLEL=3 CACHE_TYPE_K/V=q8_0` — and redeploy; no payload move |
+| 2026-10-18, 24 min (arm under the unit) | `Bonsai2-PTQ1_0/Ternary-Bonsai-2-27B-PTQ1_0.gguf` 5,946,648,928 B + hardlinked Q8 mmproj | PrismML fork 10777 | 2 × 200,704, batch/ubatch 3072/1536 | Frank scratched the 27B plan — ~48 t/s decode vs 76.6 for PQ2_0 on the same weights | `/home/dank/models/Bonsai2-PTQ1_0/` + `scripts/download-frank-zealot-bonsai2-ptq1.sh` |
+| 2026-10-17 → 10-18 | Ternary-Bonsai-2-**27B** PQ2_0 7,206,168,928 B `3907dc16…` + Q8_0 mmproj 629,246,976 B | PrismML fork **10754** (`~/GitPackages/llama-prism/`) | 2 × 200,704 (pool 401,408), q8_0 K+V = 13,328 MiB, settled 22,664 / 1,461; 76.6 t/s decode, 109.4 aggregate over two streams, 691.7 t/s prefill at 166,027; MTP impossible — no head in the file | the 9B bills half the KV per token (8 full layers, not 16), which funded three slots at the native window and beat it on every rate | `/home/dank/models/Bonsai2-PQ2_0/` + the 10754 fork tree, `scripts/download-frank-zealot-bonsai2.sh`, revert the swap commit |
+| 2026-10-07 → 10-17 | Qwen3.5-9B-Uncensored-HauhauCS-Aggressive Q6_K 7,359,259,008 B `c0ba7beb…` + BF16 mmproj 921,704,448 B | llama.cpp master `5fc4f3c` CUDA sm_86 (`~/GitPackages/llama-master/build-cuda/`) | 2 × 200,704, exact f16 K+V, MTP off (no `nextn` tensors) | the ternary 27B bought a larger model class at the same window | `/home/dank/models/Qwen35-9B-HauhauCS-Q6K/` + the master build, `scripts/download-frank-zealot.sh` |
+| 2026-10-07 (same day) | Gemma4-26B-A4B-QAT Q4_K_M 16,796,015,520 B + BF16 mmproj 1,194,827,776 B + unused `gemma4-assistant` MTP 251,937,728 B | llama.cpp master CUDA | 1 × 200,704, f16 K + q8_0 V, ubatch 1024, MTP off; 21,868 / 2,257 settled, 139.9 t/s decode (the ~4 B-active MoE is at home here); MTP ON would cost ~1.9 GiB and boot at 319 MiB free | ONE slot serialized compaction; the 9B's lighter weights bought two slots and exact KV | `/home/dank/models/Gemma4-26B-A4B/` (ungated — plain `curl -fL`) |
+| 2026-10-03 → 10-05 | MiMo-V2.6-Distill-Qwen-9B Q6_K 7,793,710,624 B `ef96d05a…` + f16 mmproj 918,166,048 B | llama.cpp master CUDA | 2 × 200,704, f16 K+V, batch 4096 / ubatch 2048; 21,846 / 2,279 settled, 94.44 t/s decode | Frank ordered the uncensored 27B with MTP drafted | `/home/dank/models/MiMo-Q6K/` |
+| 2026-10-05 (committed `77decb5`→`380aafa`, **never deployed**) | Ornith-1.5-9B Q8_0 | llama.cpp master CUDA | 1 × 262,144, f16 K+V | Swift cut over the same evening while this deploy sat blocked at the whole-tree docs gate | `/home/dank/models/Ornith-Q8/`; revert to `380aafa`'s tree |
+| 2026-10-05 → 10-07 | Swift-1.5-27B-Uncensored RCO-IQ3_XXS-mtp 10,442,827,392 B `e13d3e74…` + publisher `mmproj-BF16.gguf` 931,145,888 B | llama.cpp master CUDA | 2 × 114,688, q8_0 K+V, `draft-mtp` depth 3, ubatch 2048; 23,120 / 1,005 settled (489 MiB at the 2 × 131,072 shape — the tightest fit ever here), 63.3 t/s at the recipe temp, 72.9 at temp 0, 48.0 with the drafter off | the 9B Q6_K rung returned the 200k window and exact KV | `/home/dank/models/Swift15-UncIQ3XXS/`; the IQ3_S-mtp rung at `/home/dank/models/Swift15-IQ3S/`; commits `da36fa3` → `f2575a1` → `d8da7a9` |
+| 2026-10-02 → 10-03 | MiMo Q5_K_M 6,876,124,704 B `04cf46f2…` + the same f16 mmproj (hardlinked) | llama.cpp master CUDA | 2 × 200,704, f16 K+V, batch 2048 / ubatch 1024; 20,530 used / 4,046 apparent (true `free` 3,571), 104.84 t/s decode | the ~3.5 GiB it left free bought the Q6_K rung | `/home/dank/models/MiMo-Q5KM/` |
+| 2026-10-01 → 10-02 | MiMo Q6_K (same file as above) | llama.cpp master CUDA | 2 × 262,144, f16 K + q8_0 V | one night: the Q5 rung bought the 2–3 GiB headroom band back | `/home/dank/models/MiMo-Q6K/` (same dir) |
+| 2026-10-01 (hours) | MiMo-V2.6 INT8-W8A16-AutoRound, 8 shards 12,891,718,440 B | SGLang in venv `~/GitPackages/sglang-cu128` (torch cu130 + an in-venv CUDA 13.0 nvcc) | 2 × 262,144 over a pinned 606,208 pool, `fp8_e4m3` KV, NEXTN off; 83.9 t/s decode, ~2,050–3,370 t/s eager prefill, ~60 s boot warm | llama.cpp was the only runtime exposing asymmetric per-tensor KV | payload purged — re-fetch `old/download-frank-zealot-sglang-int8.sh`; the venv is still on disk |
+| 2026-09-17 → 10-01 | Gemma4-12B-QAT-Uncensored Q4_K_M 6.87 GiB + BF16 mmproj (it carried an **audio** encoder) + `mtp-gemma-4-12B-it.gguf` sidecar 0.24 GiB | llama.cpp master `37b53fd` CUDA with `-DGGML_CUDA_FA_ALL_QUANTS=ON` (`~/GitPackages/llama.cpp-mainline/build-cuda/`) | 4 × 262,144, q8_0 K+V, `draft-mtp,ngram-mod` depth 3 / poll 50; idle 22,346–23,300 MiB, ~965 t/s prefill and 257–302 t/s decode at 254k depth; each extra full-context slot costs 3,450 MiB so 4 is the max | the MiMo GGUF path (qwen35 hybrid, asymmetric KV) took the card | re-fetch `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced`; era trio in `git show 374041c:scripts/frank-small.service` / `.env` / `serve-frank-small.sh` |
+| 2026-09-19 (same-day revert) | `gemma-4-12B-it-qat-w4a16-ct` 10,264,229,896 B + **bf16** assistant 845,719,296 B | vLLM dev image `localhost/vllm/vllm-openai:frank-small` (`8dad46d79c46`, still in rootless storage), rootless Quadlet | 262,144 window, bf16 KV pool 289,647 tokens, 2 concurrent sessions (the 3rd queues), image + audio; 1,293 t/s prefill @67k falling to 608 @213k, ~13.6 t/s decode @213k depth, MTP acceptance 2.80–3.56, boot 135 s | reverted to llama.cpp the same day | both payloads purged; `git show 374041c^:dankclaw/frank-small/Dockerfile` and `…:dankclaw/frank-small/quadlet/frank-small.container` |
+| 2026-09-16 → 09-17 | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S + BF16 projector, in-file MTP drafted | mainline llama.cpp | single-slot 3090 shape | superseded by the Gemma4-12B payload | re-download from ISTA-DASLab — no payload and no fetch script survive |
+| 2026-09-02 → 09-15 (**Dankserver**, `:7770`, the card's previous host) | `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` ~35 GB (before it official `Qwen3.6-35B-A3B-FP8`) | FreeToken v0.1.2 (`ft`), venv `~/freetoken/.venv` | MoE cache 2,048 slots + KV 453,005 tokens + 24 mamba-state slots; ~30 t/s single-stream decode, ~87 t/s at batch 4, ~1,130 t/s prefill at 4k | the 3090 left Dankserver and `frank-medium-rocmfpx` took the endpoint and port | host units `~/.AGENTTRASH/20260915-frank-zealot-retire/host-units/`, repo trio in `scripts/.AGENTTRASH/` |
+| ~2026-08 (Dankserver 3090, pre-FreeToken) | Qwen3.6-35B-A3B Q6_K_P, then Tiel-Coder-35B-A3B UD-Q6_K_XL, with an Ornith Q6_K rung | llama.cpp `555881ebc` (b10121) CUDA | `--n-cpu-moe 38` deployed (36 recommended), batch 4096 / ubatch 2048, 700,416-token f16 K+V pool (800,768 cells did not fit); ~45.5 t/s decode and 1,188 t/s prefill at `ncmoe 35` | FreeToken kept experts in host RAM instead of CPU-offloading layers | re-download; nothing from this rung survives on disk |
+
+## Laws learned here
+
+- **Addressable ≠ advertised.** The card is 24,576 MiB but leaves **24,123 MiB**: quote `nvidia-smi`'s `free` field, never `total − used` — the driver's reservation is the difference.
+- **Exact KV and a 786,432 pool cannot coexist here**: 32,768 B/token ⇒ 24,576 MiB before a single weight loads (the retired "26,112 MiB" note doubled q8_0's scale overhead); the 27B analogue died at **25,088 MiB** with `cudaMalloc failed: out of memory`. Halving the pool funds exact KV; the per-slot window stops at the trained 262,144 (no YaRN here), so a third conversation is a KV-dtype decision, never more context.
+- **Exact KV reads 1.882× the bytes per generated token, not 2×** (32,768 vs 17,408 B/token, 2026-10-18): a decode drop of that order is EXPECTED and must be measured, while prefill should hold — and this service can afford the dtype precisely because its load is the compaction hub's ~170k-token prefill, which is weight-bandwidth bound and never re-reads the cache per generated token. The dtype is paid in decode and concurrency.
+- Never `--fit` with `parallel > 1` — it maximizes the *total* pool and shrinks each slot: set `CTX_SIZE` explicitly and keep it divisible by `PARALLEL` (the seam enforces it).
+- **KV rate law.** `full_layers × 4 KV heads × 256 dim` = **8,192 elements per token per cache** on this arch (the engine prints `n_embd_k_gqa = n_embd_v_gqa = 1024`), ×2 B (f16), ×1.0625 (q8_0 — 32 codes plus one 2-byte scale
+  per 32-block), ×0.625 (q4_0): **32,768 / 17,408 / 10,240 B/token** on this 8-full-layer 9B, and 34,816 for a
+  16-full-layer 27B at q8_0. Everything else bills separately: DeltaNet recurrent state per **slot** (150.75 MiB for
+  3 cells, and an armed depth-2 drafter triples it to 452.25), compute buffers at graph capture, the mmproj after the
+  fit. On gemma4-class hybrids `attention_k_eq_v` does **not** shrink the allocation — the cache reserves the K/V pair
+  twice, 6.125 B/elem at f16 K + q8_0 V.
+- **`--flash-attn on` is pinned, not decorative**: a quantized V exists only on the FA path and the engine falls back silently without it. llama.cpp has **no true fp8 KV dtype** — q8_0 is the 8-bit class (1.0625 B/elem, ~8.5 effective bits). `GGML_CUDA_FA_ALL_QUANTS` is vestigial and build-specific: the PrismML fork carries it ON (since 2026-09-17) while the master build had it OFF and still dispatched Q4/Q5 KV unconditionally (`ggml-cuda/fattn.cu` has no guard), so quantized KV never required it.
+- **Prices come from boots, not formulas, and an arm belongs to the payload it was measured on**: 3 × 180,224 +
+  q8_0 V fits with Q5_K_M (21,684 used / 2,441 free) and does **not** fit with Q6_K inside the 2–3 GiB band. The
+  mmproj worst case (~870 MiB) is printed before fitting but never billed by the engine's own fit check. Only
+  `tests/frank_zealot_unit.bats`'s fit gate can see a boot that would die at `cudaMalloc`, so every retune keeps it
+  green — and the first verbosity-4 ledger plus settled `nvidia-smi` of a new dtype belongs on this page, because
+  projections drift.
+- **Ubatch is a prefill knob.** Decode never moves with it and the logical `--batch-size` is neutral; prefill climbs monotonically to **2048** on this CUDA card (+4.6 % at 8.9k depth; 710 → 1,188 t/s), a prompt shorter than the ubatch cannot see the knob at all (the control row), and a verdict never crosses backends — the 2× R9700 split peaks at 512. Each rung costs ~0.6–0.75 GiB of compute buffers; 2048 is the ceiling for a 27B-on-3090 shape.
+- **A quant rung is paid in decode**: Q5_K_M → Q6_K is +875 MiB of weights and **−10 %** rate on this bandwidth-bound card, and the byte-bandwidth law predicts a rung's rate within 1 %.
+- **Speculation pays only with room on the card.** The 2026-10-18 arm census (all q8_0 K+V at the 786,432 pool): 3 × 262,144 spec-OFF ubatch 2048 → 17,876 / 6,249 free, 167.2 t/s; spec-ON ubatch 2048 → 22,706 / 1,419, 109.2 t/s; spec-ON ubatch 1024 → 21,400 / 2,725, 213.5 t/s; 2 × 262,144 spec-ON → 213.2; 1 × 262,144 spec-ON → 214.2. That is **+28 % decode whenever the card has room**, and the doors back are one window per conversation, or q8_0 KV with ubatch 1024. The draft's own bill at that shape was 3,072.00 MiB of KV plus ~1,758 MiB of buffers.
+- Draft KV is built against the target's **whole** pool (~2.4 GiB measured on the 27B Swift boot, not the ~0.4 GiB a proposal-depth estimate suggests). Spec flags move as a SET — `SPEC_TYPE=none` strips all of them and half a set has segfaulted at load. Two context-backed drafters cannot share a process, so `draft-mtp,ngram-mod` is the only legal pair, and on a qwen35 hybrid ngram-mod disables recurrent-state rollback so rejected drafts degrade silently.
+- Acceptance alone never justifies speculation: 2026-08-25 measured 26.8–35.2 t/s drafting against 53.7 no-spec.
+- A checkpoint declaring `*_num_hidden_layers` MTP while carrying zero `mtp.*`/`nextn.*` tensors boots a 4.24 GiB ghost drafter at `accept rate: 0.00` — read the safetensors index, not the config.
+- MTP is a property of the **file**: the plain PQ2_0 sibling could not draft at any flag value; only the `-MTP` export carries `blk.N.nextn.*`.
+- **The format is the runtime lock.** PQ2_0 is `general.file_type 141` over 202 tensors of custom GGML type **142** plus `prism.hadamard.*` metadata: stock llama.cpp refuses the type, and a loader ignoring the Hadamard transform would dequant **wrong, silently** (same lock class as ROCmFPX). Build ≥ b10776 is additionally the floor that pairs MTP with a quantized V cache.
+- Check arch support in the source tree: `grep -c` on the packed binary returns 0 for every arch, including the one it serves.
+- **Base alignment.** Bonsai-2 is the censored base lineage. `/home/dank/models/Bonsai2-CRACK/` matches the **27B** geometry (a one-line `MODEL_FILE` swap there, identical tensor census); no dealigned 9B exists, so for this payload the refusal escape hatch is a fetch, not a revert.
+- **A stale bench drop-in hijacks the production boot and no health check sees it.** An `Environment=FRANK_ZEALOT_ENV_FILE=/tmp/<arm>.env` in `frank-zealot.service.d/` booted a 27B PTQ1_0 payload while `/health` answered 200, `/v1/models` echoed the alias flag and the e2e gate passed 33/33 — none of them read the payload.
+- After any bench window: `systemctl show -p DropInPaths` → `tr '\0' ' ' </proc/$pid/cmdline` → ledger, in that order; drop-ins retire to `~/.AGENTTRASH/<date>-<topic>/`, and payload decisions land on this page the moment they are made, or an approved arm becomes indistinguishable from a rogue one a day later.
+- **An arm env must use the serve script's own variable names** (`CACHE_TYPE_K`/`CACHE_TYPE_V`, `BIND_ADDR`, `NGL`): an invented `KV_CACHE_TYPE` is swallowed by `: "${CACHE_TYPE_K:=f16}"` and the arm boots f16 — an OOM that reads like a model fault.
+- **Stop, shebang, GPU node.** llama.cpp rides out the default stop timeout (`TimeoutStopSec=180` is on the unit for that reason) and a hand-booted arm keeps answering 7771 behind a `failed` unit — kill it **by PID**, never `pkill -f llama-server`.
+- `serve-frank-zealot.sh` is a direct `ExecStart`, so a missing shebang is a `203/EXEC` boot loop at `Mem peak: 1.9M` with the GPU untouched: `head -1` is a deploy check.
+- `No CUDA GPUs are available` ⇒ `nvidia-modprobe -u -c=0`; KV spilled into unified memory halves decode silently, so re-read `nvidia-smi` after any shape change.
+
+## Contract
+
+- `/health` → `{"status":"ok"}`; `/props` reports `n_ctx` and `presence_penalty`; `/slots` shows occupancy; a third
+  concurrent conversation **waits** for a slot — the slot count is the concurrency contract.
+- `/metrics` answers 200 with 15 `llamacpp:*` families **only with `--metrics`**; without it llama.cpp answers **501**, which Prometheus reads as DOWN while `/health` stays 200. The scrape target is `host.containers.internal:7771/metrics` and the oldest retained sample of `up{job="frank-zealot"}` (2026-10-11) is already 0 — this target was never successfully scraped, across every payload era. `METRICS=0` is the documented opt-out and re-breaks it; the counters cost nothing on the request path.
+- Reasoning arrives in **`reasoning_content`** (`--reasoning-format deepseek`) and **counts against `max_tokens`**: a short budget yields empty `content` with `finish_reason: "length"`, which reads as a broken model, not a mid-thought one. The slot prefix cache is live (`cached_tokens` in `usage`).
+- A client that sends no sampler is **not** greedy — the engine applies the payload's own `general.sampling.*`
+  (1.0 / 0.95 / 20). `presence_penalty` is the one recipe value with no GGUF metadata key, so it must reach argv;
+  the serve script uses `${PRESENCE_PENALTY=1.0}`, deliberately **not** `:=`, because `""` is the documented opt-out.
+- Vision: only the OpenAI **`image_url`** content part reaches the model — the legacy top-level `images` field bumps `usage.prompt_tokens` while the model invents an answer. Probe with a **≥ 256 px** solid color (a postage stamp answers "video") and budget **≥ 512** tokens; a 512 px image bills ~280 prompt tokens.
+- **Q8_0 mmproj, not BF16**: the publisher warns the BF16 tower "can crash llama-server when several image requests run at once", which is exactly the `--parallel 2` shape. `MMPROJ_FILE=` (explicitly empty) is the text-only opt-out and returns ~1.4 GiB; a path at a missing file is a boot **refusal**, never silent blindness.
+- Tools: native via `--jinja`; `finish_reason: tool_calls` with valid JSON arguments.
+- Two non-faults: a `400` naming 262,144 is one conversation outgrowing its **slot**, not pool exhaustion; and
+  the raw `/completion` path is a **dead instrument on this fork build** — a greedy `ignore_eos` run answers
+  **500 "The model produced output that does not match the expected Content-only format"**, because build 10777
+  guards the raw path the moment reasoning markup leaks (older builds only logged `failed to parse reasoning
+  content` and answered anyway). The honest per-request instrument here is **`/v1/chat/completions`**, whose
+  non-stream response carries the same `timings` object (`prompt_n`, `prompt_per_second`, `predicted_n`,
+  `predicted_per_second`). Keep `VERBOSITY=4` — 3 is the engine default and cannot prove the KV or draft sizes,
+  while levels above 4 flood the journal.
+- llama.cpp does **not** validate the requested model id: it answers and echoes the served `ALIAS`, so a rename breaks id-keyed consumers (the OpenWebUI `model` row, its `access_grant`, `openai.api_configs`) and nothing server-side. Alias history: `frank-small` → `frank-secondary` (2026-10-06) → `frank-zealot` (`7c77326`, 2026-10-09); the service and unit took the name 2026-10-10 (`55f87fc`).
+- The first request after a boot is a warmup artifact (37 t/s class) and is never a measurement; never time a wall-clock aggregate on a generation that stops on its own (use `ignore_eos`).
+
+## Files
+
+- `scripts/frank-zealot.env` — **the authority for the values above** and nothing else: as of 2026-10-18 it carries
+  the knobs plus the one-line reason a reader editing that line cannot recover here. The arm table, the reference
+  boot ledger and the KV derivation all live on this page now; the header used to hold them, and two of its
+  rung figures were wrong (`CACHE_TYPE_V=q8_0` and `CTX_SIZE=401408` both free **3,840 MiB** at the 524,288 pool,
+  not 8,192 and not ~6.7 GiB — the Fallback rungs table above is the corrected statement).
+- `scripts/serve-frank-zealot.sh` — the exec line and its rationale comments; honors `FRANK_ZEALOT_ENV_FILE` for arm boots; exits 1 naming a missing binary, GGUF or projector rather than half-loading (a restart policy would turn an unfinished download into a crash loop that reads like a runtime fault).
+- `scripts/frank-zealot.service` — root unit, relinked into `/etc/systemd/system/`.
+- `scripts/download-frank-zealot-bonsai9.sh` (active) · `…-bonsai2.sh` · `…-bonsai2-ptq1.sh` ·
+  `scripts/download-frank-zealot.sh` (master-era GGUFs) · `old/download-frank-zealot-sglang-int8.sh` (SGLang).
+- `tests/frank_zealot_unit.bats` — stub-argv seam plus the VRAM fit gate.
+- `scripts/deploy-remote.sh` — `deploy_llm_zealot` (payload-path gates, MainPID-hardened health); see [[deployment]].
+- `shared/prometheus/prometheus.yml`, `shared/prometheus/rules/llm-fleet.yml`,
+  `shared/grafana/provisioning/dashboards/llm-fleet.json` — the scrape, the alert, the dashboard.
+- Live state, not repo files: the OpenWebUI picker row (back up `webui.db` before re-keying) and `~/.omp/agent/models.yml`.

serverAppConfigs-docs/LLM/freetoken-benchmark-sweep.md

@@ -90,5 +90,5 @@ Tiel-Coder finetune.
 
 ## See Also
 
-- [[freetoken-frank-small]] — production runbook
+- [[frank-zealot]] — production runbook
 - [[dealign-ornith-crack]] — the prior llama.cpp deployment (retired)

serverAppConfigs-docs/LLM/freetoken-frank-small.md

@@ -1,224 +0,0 @@
----
-type: configuration
-title: FreeToken frank-zealot Runbook
-description: RETIRED 2026-09-15 - FreeToken v0.1.2 serving batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 as frank-zealot on Dankserver (RTX 3090 era).
-tags:
-  - freetoken
-  - dankserver
-  - frank-zealot
-  - qwen3.6
-  - moe
-  - fp8
-timestamp: 2026-08-30
----
-
-# FreeToken frank-zealot Runbook
-
-> **RETIRED 2026-09-15.** The RTX 3090 left Dankserver and the successor
-> [[frank-medium-rocmfpx]] (charlie12345/ROCmFPX llama.cpp on 2x Radeon AI
-> PRO R9700) took the :7770 endpoint and the port. This page is kept as the
-> retirement record of the FreeToken/CUDA era; the unit was archived to
-> `~/.AGENTTRASH/20260915-frank-zealot-retire/host-units/` and the repo files
-> to `scripts/.AGENTTRASH/`.
-
-The `frank-zealot` service on Dankserver is served by
-**FreeToken v0.1.2** — an edge-native MoE serving engine that keeps
-expert weights in host RAM with an LRU GPU cache. Model history on this
-runtime: official `Qwen/Qwen3.6-35B-A3B-FP8` (since 2026-09-02) →
-`batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` (2026-08-30,
-same format and pool budget; re-landed 2026-09-01 after two
-compressed-tensors FP8 candidates failed to load — see Quantization
-Compatibility). The engine itself replaced the llama.cpp
-Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment.
-
-## Model
-
-| Field | Value |
-|-------|-------|
-| HuggingFace | `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` |
-| Lineage | Qwen/Qwen3.6-35B-A3B → huihui-ai abliterated → this FP8 (quantizer batsclamp) |
-| Architecture | `qwen3_5_moe` (Qwen3_5MoeForConditionalGeneration) |
-| Quant | native modelopt block-fp8 e4m3, dynamic activations, 128×128 `weight_block_size`, bf16 `weight_scale_inv` — byte-format identical to the official FP8 repo |
-| BF16 kept | lm_head, embed_tokens, all norms, router gates, mamba state params, visual encoder, MTP module |
-| Experts | un-fused per-expert layout (`mlp.experts.{0..255}.*`) matching official Qwen FP8 — same FreeToken loader path |
-| Size on disk | ~35 GB (26 safetensors shards) |
-| Path | `/home/dank/models/huihui-qwen36-35b-a3b-abliterated-fp8` |
-| Layers | 40 (10 full-attention + 30 GDN/linear-attention) |
-| Experts | 256 per layer, 8 used per token |
-| KV heads | 2 × 256 dim |
-| Context | 262,144 (model max) |
-| Chat template | froggeric/Qwen-Fixed-Chat-Templates (fixed Qwen3.6 template, shipped in `tokenizer_config.json`; FreeToken reads it from the checkpoint — no override) |
-
-**Important:** this is the Huihui *abliterated* finetune (refusals
-removed) — not the base Qwen weights served before 2026-08-30, and not
-the earlier Tiel-Coder finetune. Expect a direct-answers assistant.
-
-## Runtime
-
-| Field | Value |
-|-------|-------|
-| Engine | FreeToken v0.1.2 (`ft` CLI) |
-| Venv | `/home/dank/freetoken/.venv` (uv-managed CPython 3.12) |
-| Dependencies | torch 2.11.0+cu130, flashinfer 0.6.17, apache-tvm-ffi |
-| CUDA toolchain | pip-installed nvidia-cuda-nvcc/crt/nvvm/runtime 13.2.86 |
-| API | OpenAI-compatible (`/v1/chat/completions`, `/v1/models`) |
-
-## Pool Configuration
-
-The GPU budget (~17.6 GiB after weights + fixed caches) is split
-MoE-first between expert residency and KV pages:
-
-| Pool | Size | VRAM | Notes |
-|------|------|------|-------|
-| MoE cache | 2,048 slots (20%) | 6.0 GiB | LRU expert cache, out of 10,240 total |
-| KV cache | 453,005 tokens | 8.6 GiB | ~20 KiB/token (10 attn layers only) |
-| Mamba state | 24 slots | 1.4 GiB | Per-sequence GDN state (caps concurrency) |
-
-### Live Resize
-
-KV and MoE can be re-partitioned without restart:
-
-```bash
-ft ctl --base-url http://127.0.0.1:7770 cache --kv N --moe M --wait 300
-```
-
-- **Growing KV** (shrinking MoE): works live.
-- **Shrinking KV** (growing MoE): requires restart (old + new pools
-  cannot coexist in VRAM during the swap → OOM).
-- The fit check refuses overcommit: if the plan doesn't fit, the old
-  cache is retained and serving continues.
-
-### Benchmark Results (2026-09-02, base FP8 checkpoint)
-
-See [[freetoken-benchmark-sweep]] for the full bs1/bs3/bs4/bs6 sweep
-across KV pool sizes 8k–700k.
-
-At the production config (20% residency, bs4):
-
-| Metric | Value |
-|--------|-------|
-| bs1 decode | ~30 tok/s |
-| bs4 aggregate | ~87 tok/s |
-| bs4 per-request | ~22 tok/s |
-| Prefill (4k) | ~1,130 tok/s |
-
-## Quantization Compatibility
-
-FreeToken v0.1.2 supports these safetensor quants for `qwen3_5_moe`:
-
-| Format | Detection | Works |
-|--------|-----------|-------|
-| BF16 (no quant config) | default | ✅ (experts too large for 60 GB RAM) |
-| modelopt block-fp8 (128×128) | `quant_method: "fp8"` + `weight_block_size` | ✅ (current) |
-| NVFP4 (modelopt) | `quant_algo: "NVFP4"` | ✅ (best for VRAM) |
-| NVFP4 (compressed-tensors) | 4-bit float, group_size=16, strategy=tensor_group | ✅ |
-| Per-tensor FP8 (modelopt) | scalar `weight_scale` | ✅ |
-| Per-channel FP8 (compressed-tensors) | `weight_scale: [O, 1]` | ❌ loader bug |
-| Block-fp8 128×128 (compressed-tensors) | `quant_method: "compressed-tensors"` + `block_structure: [128, 128]` | ❌ dequant shape bug |
-| Quark W8A8 INT8 | per-channel scales | ❌ loader bug |
-| GGUF | only `gemma4` arch | ❌ for qwen3_5_moe |
-
-**2026-09-01 live verification (Dankserver):** both compressed-tensors
-FP8 shapes fail on FreeToken v0.1.2, each a clean exit at load (37 GB
-fits RAM — no swap-thrash). `ornith-ai/Ornith-1.5-35B-A3B-FP8`
-(per-channel) aborts in expert-source resolution:
-`ValueError: Missing MoE expert source layers: {'gate_up': ...}` — its
-expert tensors do not follow the `mlp.experts.{N}.gate_proj/up_proj/
-down_proj` naming the MoE provider scans for.
-`lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8` (block
-128×128, standard naming) resolves experts but aborts in dequant:
-`RuntimeError: The size of tensor a (2048) must match the size of
-tensor b (...)` in `_dequant_fp8_weight`, which multiplies
-`weight × weight_scale` directly instead of expanding the block scales.
-modelopt block-fp8 remains the only FP8 path that loads.
-
-## Sampling
-
-**frank-zealot samples by default, and our config never said so.**
-`scripts/serve-frank-zealot.sh` passes no sampling arguments at all — only
-`--model`, `--served-model-name`, `--host`, `--port`, `--moe-cache-size`,
-`--num-tokens` and `--max-running-requests`. The behaviour comes from the
-runtime's own default:
-
-```python
-# freetoken/server/args.py:31
-sampling_defaults: str = "model"
-```
-
-`--sampling-defaults model` fills `temperature` / `top_k` / `top_p` from the
-**checkpoint's** `generation_config.json` for any request that omits them.
-This checkpoint ships:
-
-```json
-{ "do_sample": true, "temperature": 1.0, "top_k": 20, "top_p": 0.95 }
-```
-
-Verified behaviourally 2026-09-12 — three identical requests with **no
-sampling parameters sent at all** produced three different reasoning traces
-(sha `13ca02…`/487 chars, `d49928…`/540, `7b81b8…`/577). Greedy decoding
-would be byte-identical.
-
-### This is the opposite of frank-dragoon
-
-| | frank-zealot (FreeToken) | frank-dragoon (halogen) |
-|---|---|---|
-| Silent client (no `temperature`) | **samples** at temp 1.0 / top_k 20 / top_p 0.95 | **greedy** — no server-side sampler defaults |
-| Deterministic across runs | No | Yes |
-| Where the default comes from | checkpoint `generation_config.json` | none; the request decides |
-
-A client that talks to both endpoints and sends no sampler gets
-non-deterministic answers from :7770 on Dankserver and reproducible ones from
-:7770 on Dankclaw. If a workflow needs determinism from frank-zealot it must
-send `temperature: 0` explicitly, or the service must be relaunched with
-`--sampling-defaults none` (framework defaults only).
-
-### Related request behaviour
-
-- **Thinking consumes the budget.** At `max_tokens: 200` all three probe runs
-  returned `finish_reason: "length"` with **empty `content`** and the whole
-  budget spent in `reasoning_content`. Budget for thinking, or the answer
-  never appears.
-- **The API does not validate unknown fields** — a request carrying
-  `totally_bogus_field: 123` returns 200, as does
-  `chat_template_kwargs: {"preserve_thinking": true}`. A 200 is not evidence
-  that a key was understood.
-
-
-## Files
-
-| File | Purpose |
-|------|---------|
-| `scripts/serve-frank-zealot.sh` | Serve script (launches `ft serve`) |
-| `scripts/frank-zealot.env` | Tunables (model path, pool sizes) |
-| `scripts/frank-zealot.service` | Systemd unit |
-
-## Limitations
-
-- **Model identity**: serves the Huihui abliterated FP8 finetune, not the Tiel-Coder coder finetune from the llama.cpp era.
-- **JIT compilation**: first startup compiles flashinfer/tvm-ffi kernels
-  (~5–15 min cold). Cached afterward in `~/.cache/flashinfer` and
-  `~/.cache/tvm-ffi`.
-- **CUDA toolchain**: driver-only install (no system CUDA toolkit). The
-  venv provides pip-installed nvcc 13.2.86 for JIT compilation. All
-  nvidia-cu13 packages must be at the same version.
-- **Env var collisions**: never name tunables `TEMP`, `HOST`, or
-  `GPU_MAX_HEAP_SIZE` (HIP runtime reads these).
-
-## Operational quirks
-
-- **`model` field is required** in `/v1/chat/completions` bodies —
-  FreeToken rejects requests without it with a 422 (`Field required`),
-  unlike llama.cpp's server which defaults to the loaded model. Send
-  `"model": "frank-zealot"`.
-- Load on this swap took well under the old llama.cpp numbers:
-  weights → experts (parallel build) → CUDA graph capture → Uvicorn
-  in ~2 minutes; the 2026-08-30 swap verified ~40 tok/s decode on a
-  282-token completion.
-
-## Citations
-
-- FreeToken paper: arXiv 2608.16157
-- FreeToken PyPI: `freetoken[accel]` v0.1.2
-- HuggingFace model: `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8`
-- Upstream weights: `huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated`, `Qwen/Qwen3.6-35B-A3B-FP8`
-- Fixed chat template: `froggeric/Qwen-Fixed-Chat-Templates`

serverAppConfigs-docs/LLM/halogen-flash-server.md

@@ -1,478 +0,0 @@
----
-type: reference
-title: halogen-flash-server on Dankclaw (frank-dragoon)
-description: The active frank-dragoon runbook — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing.
-tags:
-  - llm
-  - halogen
-  - dankclaw
-  - frank-dragoon
-  - strix-halo
-  - gfx1151
-  - qwen4exp
-  - rocm
-timestamp: 2026-09-13
----
-
-# halogen-flash-server on Dankclaw (frank-dragoon)
-
-> **ACTIVE again since 2026-09-16, and retuned.** It replaced the llama.cpp /
-> kingjones30-ROCmFPX STRIX deployment on 2026-09-11 (see
-> [[kingjones-strix-rocmfp4]]), was displaced for two days by ik_llama.cpp (see
-> [[frank-dragoon-ik-uncensored]], the current rollback), and came back as the
-> shape the fleet asked for: two 200k-token conversations rather than the
-> native-context pair. Deployed as a rootless Quadlet container,
-> `dankclaw/frank-dragoon/quadlet/frank-dragoon.container`, port 7770. The RTX 3090
-> it does not touch now belongs to [[frank-zealot-qwen38-gsq-rco]].
-
-The engine is a from-scratch serving runtime for the Qwen3.8-Flash-Next
-architecture on AMD's gfx1151 (Strix Halo) — the same silicon this host runs.
-Upstream's own comparison table measures it at **roughly 4x the prefill
-throughput** of the llama.cpp-family runtimes it ships against (EngramHalo.cpp,
-ROCmFP4, CIRU-IU4) on a 32K prompt, and prefill is most of the wall clock on any
-prompt with real context. That, not decode, is the reason for the swap.
-
-## Shape
-
-One container runs both halves of the stack:
-
-| half | port | reachable |
-|---|---|---|
-| engine | 8730 | container loopback only — **unauthenticated, never publish it** |
-| OpenAI front-end | 8731 | published as host **7770** |
-
-The `all` mode is upstream's default and the shape a single-container host
-wants: the front-end talks to the engine over loopback inside the container, so
-there is one process tree, one log, and one published port.
-
-`HALOGEN_MODEL_ID=frank-dragoon` sets the id reported at `/v1/models`, in
-`/health`, and in every response body. Requests are **never rejected for naming
-a different model** — the server serves whatever asks. Keeping the fleet alias
-is therefore free, and it means Hermes, the harness config, the OpenWebUI
-discovery, the `services.json` probe and the docs needed no rename.
-
-## Weights
-
-`peonist-ai/halogen-qwen3.8-flash-next` (HuggingFace, public and ungated):
-
-| file | size | role |
-|---|---|---|
-| `qwen38-flash-next-w4b.hgn` | 115.55 GiB | the checkpoint |
-| `qwen38-flash-next-w4b.overlay.hgn` | 2.40 GiB | **quality sidecar** — picked up automatically; the engine warns at startup if it is missing. **Refreshed 2026-09-16**: the sidecars are versioned *in place*, so a stale local copy is invisible from its file name — this one had read 2.31 GiB since it was first fetched while the Hub had moved on |
-| `qwen38-flash-next-vision.hgn` | 0.84 GiB | vision tower (see below) |
-| `tokenizer/` | ~21 MB | flat tokenizer dir; found beside the checkpoint, so no extra mount |
-
-`HALOGEN_DOWNLOAD` fetches the whole repo into the bind mount on **first start
-only** — the entrypoint acts only when the checkpoint is absent, so a restart
-never re-downloads, and `huggingface_hub` resumes an interrupted pull. The
-mount must be read-write for this. Roughly 127 GB; on Dankclaw it lands in
-`/home/dank/models/halogen`.
-
-**Refreshing the weights.** Delete the mount's contents (or the whole
-directory, then recreate it **empty**) and start the unit: the entrypoint sees
-the checkpoint absent and re-fetches the repo, resuming if interrupted. Two
-traps. First, the bind mount's source directory must *exist* — podman refuses
-the container with `Error: statfs /home/dank/models/halogen: no such file or
-directory` and the unit loops on `Restart=on-failure`; `mkdir -p` the empty
-directory and the fetch starts on the next start. Second, "refresh" only means
-"fetch what the Hub has now": on 2026-09-16 the re-fetch brought newer sidecars
-(`overlay.hgn` 2,477,677,120 → **2,572,466,560** bytes, `overlay-speed.hgn`
-2,383,306,048 → **2,478,095,488**) while the checkpoint returned
-byte-identical in size (124,068,083,904) — so an update can live entirely in
-the sidecars, and equal file sizes are not proof of an unchanged repo.
-
-`overlay-speed.hgn` (2.31 GiB on the Hub; 2.22 GiB before the 2026-09-16
-refresh) comes along with the repo but is unused: it trades the `o_proj`
-calibration for about 2 % of decode, and the quality sidecar is the default for
-a reason.
-
-## Deployed flags
-
-Everything else is configured by environment variable — there is no config
-file, and the flags are read once at startup. These are the departures from
-upstream's defaults:
-
-| flag | value | why |
-|---|---|---|
-| `HALOGEN_MODEL_ID` | `frank-dragoon` | fleet identity; no client rename |
-| `HALOGEN_CTX` | `200192` | **set 2026-09-16**: exactly half the pool, so two conversations of the full window are always resident. Not a round 200,000 — pool/2 is. The model's native window is 262,144, so this is a real reduction in the per-conversation ceiling, and a client asking for more gets a 400 (never a silent truncation). The harness declares 200192 for `frank-dragoon` |
-| `HALOGEN_KV_POOL_POSITIONS` | `400384` | **retuned 2026-09-16** (from 401,408): 2 x 200,192. Reserves ~11.0 GiB of KV, confirmed by the boot ledger as `400384 positions, ~27.2 GiB (16.2 GiB of it for everything but the pool, at max_tok 24576)` against a 33.5 GiB device budget — **no clamp**. Slots still oversubscribe the pool for any real request (admission reserves prompt + `max_tokens`), so a near-full pair that is both generating is pool-bound. `HALOGEN_KV_POOL_FIT` guards the boot: read the ledger, not this column |
-| `HALOGEN_KV_SLOTS` | `2` | **lowered from 3 on 2026-09-16** — one slot per half of the pool, and two is the fleet's shape now that Dankclaw serves two endpoints. Each slot is 111.0 MiB of O(1) state, so this hands ~111 MiB back as well |
-| `HALOGEN_MAX_TOK` | `24576` | **halved from the 32768 default on 2026-09-16** for frank-zealot, its co-tenant on the same host. The arena's variable term goes `8.40 x 2 → 8.40 x 1.5`, handing ~4.2 GiB back at roughly a 4-5 % prefill cost — the same lever at 16,384 costs 8-9 %. The vendor's own notes call 24,576 the largest arena that allocated with headroom to spare, so it stays inside their tested range. 65536 still cannot boot. It sizes a per-call PREFILL CHUNK, not the prompt ceiling: a 200k conversation is prefilled in arena-sized pieces |
-| `HALOGEN_HOST_RESERVE_GIB` | `20` | **raised from 15 with the same retune**, because the arena shrank: `budget = 121.2 − 67.7 − 20 = 33.5 GiB` against a measured need of ~27.2, so the fit check kept ~6 GiB of margin and did not clamp. It is **not** an allocation: raising it cannot return memory to the host, only take the pool away (see the ceiling section below) |
-| `HALOGEN_VISION_TOWER` | *unset* | **removed 2026-09-16.** The tower charges 1.97 GiB against the same device budget as the pool and the arena, and the fleet's image path moved to [[frank-zealot-qwen38-gsq-rco]] (llama.cpp with `--mmproj`). This server refuses image content parts, and `models.yml` declares `frank-dragoon` text-only to match |
-| `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch |
-| `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug |
-
-Everything else is upstream's default, deliberately: prompt cache mode `2`
-(two entries per conversation — end-of-system-prompt and end-of-history — which
-already covers the shared-prefix case), `HALOGEN_CACHE_INPLACE=1` (the snapshot
-stays ~115 MiB at any depth), the speculative drafter `1` (byte-identical at
-temperature 0, and it stands down whenever a second conversation is
-generating), the quality sidecar, `HALOGEN_ADMIT_CHUNK=0` (a non-zero value
-gives up the byte-identity property), and the default `HALOGEN_MAX_TOKENS_CAP=65536`.
-
-## Memory model
-
-Three numbers explain the whole box:
-
-- **~68 GiB pinned weights.** The checkpoint is mapped and registered with the
-  GPU in place, never copied. The kernel counts the locked pages as reclaimable
-  file cache, so `free`, `MemAvailable` and every dashboard built on them
-  **overstate free memory by the size of the model**. The server prints the
-  corrected figure at startup; believe that line, not Grafana.
-- **~28 KiB per KV position** (measured: 262,144 positions = 6,656 MiB of KV,
-  ~7.2 GiB with block scratch), plus the prefill arena. The 2026-09-16 ledger
-  reads the two together in the engine's own words: `400384 positions, ~27.2
-  GiB (16.2 GiB of it for everything but the pool, at max_tok 24576)` — so
-  ~11.0 GiB of pool and 16.2 GiB of arena, working slack and (no longer) a
-  vision tower. The engine's device-side budget stops at roughly 47 GiB on a
-  128 GB machine *with the weights pinned* — and lower on a machine carrying
-  more besides this server, which Dankclaw is.
-- **`HALOGEN_HOST_RESERVE_GIB`** is what is left for the page cache the lookup
-  table is read through. Too little and long prompts crawl with the disk busy
-  and the process in uninterruptible sleep — that is a cache problem, not a
-  memory problem, and the fix is a *smaller* pool, not a bigger one.
-
-`HALOGEN_KV_POOL_FIT=1` (default) measures that budget at startup and lowers
-the pool itself if the configured one will not fit, printing what it chose.
-Read those lines on every restart; they are the ground truth for sizing.
-
-**Measured ledger, seven restarts, read live 2026-09-12.** The engine's own
-startup account, including three deliberate experiments:
-
-| pool | max_tok | reserve | KV | arena | need | budget | verdict |
-|---|---|---|---|---|---|---|---|
-| 262144 (baseline) | 32768 | 24 | 7.2 | 20.6 | 29.8 | 29.5 | boots 0.3 over — prints **no** sizing lines |
-| 524288 | 16384 | 24 | 14.4 | 12.2 | 28.6 | 29.5 | fits, 0.9 spare |
-| 524288 | 32768 | 24 | 14.4 | 20.6 | 37.0 | 29.5 | clamped to 262144, deploy still green |
-| 524288 | 65536 | 24 | — | 37.3 | 46.5 | 29.5 | **OOM — will not boot** |
-| 524288 | 32768 | **15** | 14.4 | 20.6 | 37.0 | **38.5** | **deployed — both features** |
-| 524288 | 32768 | 16 | 14.4 | 20.6 | 37.0 | 37.5 | fits, footprint unchanged — see the ceiling below |
-| **400384** | **24576** | **20** | **11.0** | **16.4** | **27.2** | **33.5** | **deployed 2026-09-16 — ledger need 27.2, no clamp** |
-| 401408 | 32768 | 15 | 11.0 | 20.6 | 33.4 | 38.5 | 2026-09-13 → 09-16, superseded by the retune |
-
-### The sizing rule
-
-One budget covers the pool, the arena **and** the vision tower together:
-
-```
-device budget = host RAM - resident weights - HALOGEN_HOST_RESERVE_GIB
-              = 121.2    - 67.7             - reserve
-                                    reserve 24 -> 29.5 GiB
-                                    reserve 15 -> 38.5 GiB
-
-need = KV (28 KiB/position) + arena + vision (1.97 measured)
-arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384)
-```
-
-The `121` the engine prints is rounded **down** from `MemTotal 127104060 kB =
-121.2159 GiB`, so subtracting the displayed figures gives 29.3 rather than 29.5.
-Use the unrounded value. Two of the three terms are fixed hardware and
-checkpoint facts; `HALOGEN_HOST_RESERVE_GIB` is the only one you own — which
-buys the *fit margin*, not the host's memory. It is a term in this inequality,
-not an allocation, and the section at the end of this page shows the live test
-that proves it.
-
-Trust the engine's printed requirement over this model wherever they disagree:
-a boot clamped at reserve 24 stated `524288 positions need ~35.0 GiB (plus 1.5
-of margin)`, i.e. **budget >= 36.5**, against a modelled 37.0 here. Close
-enough to plan with, and the authoritative number to cut the other way.
-
-The arena term is **linear in `HALOGEN_MAX_TOK`** and predicts every measured
-point to 0.1 GiB — including upstream's three published "on the device" figures
-(524,288 → ~35, 786,432 → ~42, 1,048,576 at `max_tok=16384` → ~41), which are
-**pool + arena totals at the default `max_tok`**, not pool alone. That is why
-their largest pool "needs `HALOGEN_MAX_TOK=16384`": at 32768 a 1M pool would be
-49.4. Reading those figures as pool-only is the trap that makes a doubling look
-affordable when it is not.
-
-At the baseline reserve of 24 the doubled pool leaves `29.5 - 1.97 - 14.4 =
-13.1` for the arena, so `max_tok <= 18,130` and **16384 is that reserve's
-ceiling, not a conservative setting.** The corner check agrees: at 32768 only
-7.0 GiB is left for the KV, ~254k positions, and the pool drops back toward
-262144. Cut the reserve to 15 and the same arithmetic gives `38.5 - 1.97 - 14.4
-= 22.1` for the arena — `max_tok <= 35,700`, which is exactly why 32768 now
-fits unmodified.
-
-### What the reserve actually costs
-
-The device budget is not the whole story. Memory position by configuration
-(401,408 is the 2026-09-13 deployed row, read from its boot ledger):
-
-| config | held | host left | contiguous 2 MiB blocks |
-|---|---|---|---|
-| 262144 / 32768, reserve 24 | 96.3 | 12.1 GiB | 2081 |
-| 524288 / 16384, reserve 24 | 94.6 | 13.8 GiB | 3742 |
-| 524288 / 32768, reserve **16** | **103.5** | **7.1 GiB** | 293 |
-| **401408 / 32768, reserve 15** | **100.0** | **9.1 GiB** | **1396** |
-| **400384 / 24576, reserve 20** | **95.4** | **14.7 GiB** | **475** |
-
-At the doubled setting (2026-09-12) the server held 103.5 of 121 GiB and the
-engine volunteered the stall note. It did not print it at the 401,408 trim, and
-it prints it again after the 2026-09-16 retune. Three data points, and they do
-not line up with a "free memory below X" reading: 310 blocks printed, 1396 did
-not, 475 did. So treat the note as "the machine is tight" and read the
-`host memory left` line rather than waiting to be told:
-
-```
-halogen: NOTE this server now holds most of the machine, and 310 contiguous
-2 MiB blocks remain. It runs best on a host of its own. If other large
-processes share this machine they will compete for what is left, and when it
-runs out BOTH they and this server can stall for minutes at a time at full
-CPU with no disk activity and no output, which is not a crash and needs no
-restart.
-```
-
-That is the failure mode to watch, and it is the least diagnosable of the
-three: not a silent clamp, not a boot refusal, but a minutes-long freeze that
-looks like a hang, for the neighbours as much as for frank-dragoon. `free` is
-useless for spotting it — `MemAvailable` still reads ~74 GiB because the
-page cache counts as reclaimable. Watch the `host memory left for everything
-else` line and the contiguous-block count at boot instead.
-
-28 co-tenant containers (~4.3 GiB working set) stayed up through the change,
-so it is survivable.
-
-### The reserve cannot buy air — ceiling 16
-
-The last row above is the experiment that proves it. Raising the reserve from
-15 to 16 left `held` and `host left` **byte-for-byte identical** (103.5 / 7.1
-GiB). `HALOGEN_HOST_RESERVE_GIB` feeds the fit-check budget and is *not* an
-allocation; the actual footprint is `weights + KV + arena + vision`, none of
-which the reserve touches. So the knob has exactly two regimes:
-
-| reserve | budget | vs required 36.5 | effect |
-|---|---|---|---|
-| 15 | 38.5 | +2.0 | deployed |
-| 16 | 37.5 | +1.0 | no change to footprint, thinner margin — pointless |
-| 17 | 36.5 | +0.02 | knife edge, inside the noise of the 67.7 term |
-| **18** | **35.5** | **−1.0** | **clamps the pool to 262144** |
-| 24 | 29.5 | −7.0 | clamps the pool to 262144 |
-
-The 36.5 requirement is the engine's own, read from a clamped boot:
-`524288 positions need ~35.0 GiB (plus 1.5 of margin)`. Above 16 the only
-thing a bigger reserve can do is take the doubling away.
-
-**To actually free host memory, change the pool or the arena**, and the reserve
-then becomes free to rise: `HALOGEN_KV_POOL_POSITIONS=262144` hands back
-7.2 GiB from the doubled pool, `HALOGEN_MAX_TOK=16384` hands back 8.4. Both
-are one-line edits that the fit check will confirm on the next boot. The
-2026-09-13 trim to 401,408 was this remedy applied at half strength —
-`HALOGEN_MAX_TOK` untouched, and on the host-left metric it bought +2.0 GiB
-with a 4.5x improvement in contiguous blocks (310 → 1396), less than the
-modelled 3.4 because the engine holds the freed KV as working slack.
-
-### The `WARNING` line only appears when the fit check had to move
-
-This one cost me a wrong conclusion, so it is worth stating plainly. When the
-configured pool already fits, the engine reserves it **and prints nothing** —
-the 09-11 baseline boot goes straight from the pre-flight line to
-`checkpoint: mapped` with no `kv pool:` sizing block at all. The sizing block and
-its `WARNING even 262144 positions ... host RAM cannot spare it` appear **only
-after a clamp**.
-
-So the absence of a warning in old logs is not evidence of headroom. Both
-`262144/32768` rows above need 29.8 against a 29.5 budget — 0.27 over — and the
-baseline has lived in that state since it was deployed without incident,
-because the 24 GiB reserve is itself generous (the host reports 12–14 GiB free
-at that point). The engine's budget is a conservative floor, not a hard wall.
-
-### Two failure modes, and only one is dangerous
-
-Raising `max_tok` past the ceiling does not shrink the pool — the arena alone
-outgrows the budget, and the server cannot allocate at all:
-
-```
-kv pool: 524288 positions need ~51.7 GiB ... LOWERING THE POOL TO 262144 (~44.5 GiB).
-kv pool: WARNING even 262144 positions (the context, the smallest pool that can
-         serve one request) needs ~44.5 GiB and host RAM cannot spare it.
-dmalloc: FAILED requesting 0.250 GiB after 39.970 GiB in 566 allocations (out of memory)
-```
-
-This is the quiet one: the unit stays `active`, the container reports
-`Up (starting)`, and `deploy.sh` sits in its health wait for the whole
-`LLM_WAIT` rather than failing fast. It confirms upstream's warning that a
-large `HALOGEN_MAX_TOK` means "the server will not start", not "it will be
-slower".
-
-Under-running the ceiling is far more benign and far easier to miss. At 524288
-with the arena back at 32768, `HALOGEN_KV_POOL_FIT` lowered the pool and
-`deploy.sh` still reported healthy in 12 seconds:
-
-```
-kv pool: 524288 positions need ~35.0 GiB (plus 1.5 of margin) and host
-         RAM cannot spare it. LOWERING THE POOL TO 262144 (~27.8 GiB).
-```
-
-A green health check does not mean the pool you asked for is the pool you got —
-read the `kv pool:` lines after every restart.
-
-The 47.7 GiB n-gram lookup table is never held in RAM; it is read through
-the file cache. The 2.0 GiB the firmware carves for the iGPU appears in no
-`/proc/meminfo` figure at all.
-
-**Co-tenancy.** `HALOGEN_FLASH_PIN_TRUNK=1` (default) is what the published
-decode numbers were measured with, and it means this container holds most of
-the host. If a co-tenant then takes the remainder, both it and the engine can
-stall for minutes at a time at full CPU with no disk activity and no output.
-That is documented behaviour, not a crash, and it needs no restart.
-`HALOGEN_FLASH_PIN_TRUNK=0` gives the memory back and costs several times the
-decode speed — a last resort, not a tuning option.
-
-## Client contract
-
-The differences from llama.cpp that bite are all in the request, not the
-deployment:
-
-- **No server-side sampler defaults.** `temperature` absent or `0` is
-  **greedy** decode; above 0 the request samples on the same drafter. Supported
-  fields: `temperature`, `top_p`, `top_k`, `min_p`, `seed`,
-  `presence_penalty`, `frequency_penalty`, `logit_bias`, `logprobs`. Refused
-  with a 400: `top_logprobs`, `logprobs` with `stream: true`, `n > 1`, and any
-  value outside its defined range (values are refused, never clamped).
-  The fleet runs greedy for clients that stay silent — deterministic, and
-  byte-identical with the drafter.
-- **The token budget covers thinking.** The default is **8192**, and a budget
-  that runs out mid-thought returns `finish_reason: "length"` with an **empty
-  `content`** and the partial reasoning in `reasoning_content` — which most
-  clients do not display. Above the cap (`HALOGEN_MAX_TOKENS_CAP`, 65536) a
-  request is a 400 rather than a silent truncation. `max_tokens`,
-  `max_completion_tokens` and `max_output_tokens` all work and must agree if
-  more than one is sent.
-- **A budget above (window − compaction-threshold) is a guaranteed 400 zone.**
-  The fit check (`prompt + max_tokens <= HALOGEN_CTX`) refuses, and the harness
-  only compacts at its own threshold — with OMP at 70 % of a 200,192 window
-  (140,134), any prompt in (136,192, 140,134] with a 64,000 budget overflows
-  the window while sitting below the compaction trigger. That collided live on
-  2026-09-17 (prompt 138,898 + 64,000 = 202,898 > 200,192). The fix is
-  harness-side: keep the declared `maxTokens` at or under
-  `window − threshold` (57,344 chosen) so no pre-compaction request can miss
-  the fit check. Do not fix it by raising `HALOGEN_CTX` past pool/2 — that
-  breaks the two-resident-conversations property.
-- **`reasoning_effort`** is accepted at the top level and inside
-  `chat_template_kwargs`: `minimal`, `low`, `medium`, `high`, `xhigh`. The
-  model's default is `xhigh`. `minimal` aliases the template's `low` and still
-  reasons; `enable_thinking: false` turns thinking off; `preserve_thinking` is
-  honoured. `preserve_reasoning` is a **400** — it is the llama.cpp-era key.
-- `/v1/models`, `/health` and completions all report the served id, and
-  `reasoning_content` carries the reasoning trace.
-
-**Pool admission is the one arithmetic trap.** A request reserves its prompt
-**plus** `max_tokens` positions when it is admitted; if the pool cannot hold it
-the request waits in arrival order, and after the queue timeout (3600 s) it
-gets a `503 engine_busy`. A prompt large enough that prompt + budget exceeds
-the pool can therefore never be admitted at all. The 2026-09-16 shape is a
-400,384-position pool with `HALOGEN_CTX` 200,192 and two slots: two
-conversations at the ceiling fit the pool exactly, so a pair that is both
-near-full and both generating is pool-bound — the second waits for the first to
-release its region. The pool binds **how many** conversations stay resident,
-not how long one may be; the older 524,288 / 401,408 numbers live in the ledger
-tables above.
-
-## Vision
-
-**None here since 2026-09-16.** `HALOGEN_VISION_TOWER` is deliberately unset:
-the tower is a 1.97 GiB charge against the same device budget as the pool and
-the arena, and the fleet's image path moved to [[frank-zealot-qwen38-gsq-rco]],
-which loads the BF16 projector alongside its own weights. This server refuses
-image content parts and says why, rather than ignoring them:
-
-```json
-{"error": {"message": "this server does not accept images: the engine was
-started without a vision tower (HALOGEN_VISION_TOWER)", "type":
-"invalid_request_error"}}
-```
-
-The llama.cpp side handles images the same way this engine used to: a `data:`
-URL or bare base64, about 2,040 tokens of context per image, and `http(s)` URLs
-refused deliberately (fetching one would make the server issue outbound
-requests to wherever a client pointed it). Audio and video have no path
-anywhere in the fleet — that left with [[frank-npu]].
-
-## Deploy and operate
-
-```bash
-./scripts/deploy.sh dankclaw --service frank-dragoon
-```
-
-The handler retires the ik-era **root** unit first (stop, disable, move
-`/etc/systemd/system/frank-dragoon.service` into
-`~/.AGENTTRASH/20260916-halogen-restore/host-units` — never `rm`; this host's
-`disable` may delete the file outright, which the handler now reports), stops
-the Quadlet's own unit if it is running, then pulls
-`ghcr.io/peonist-ai/halogen-flash-server:0.5.8`, re-tags it to the
-`localhost/` alias the unit references, and **waits for port 7770 to be free**
-before installing and starting anything.
-
-That last gate is not paranoia. On the first cutover attempt the ik server
-logged "Deactivated successfully", and pasta — rootless podman's userspace
-network stack, which binds the published port at container CREATE — still
-reported `Address already in use` a second later, exiting the unit with code
-126. Only `Restart=on-failure` recovered it, minutes later. A deploy that
-starts a doomed container has not deployed anything, so `wait_port_free` polls
-every socket on the port (listener, accepted, TIME_WAIT) and fails loudly with
-the socket table if it never clears.
-
-Boot persistence comes from `[Install] WantedBy=default.target` in the unit
-file plus linger on the deploy user — generated units **cannot** be
-`systemctl --user enable`d on this host.
-
-```bash
-systemctl --user status frank-dragoon.service     # the generated unit
-podman logs frank-dragoon | head -40              # version banner + memory ledger
-curl -s http://127.0.0.1:7770/health            # 200 only after the engine PONGs
-curl -s http://127.0.0.1:7770/v1/models
-```
-
-`/health` asks the **engine** for a PONG before answering, so a 200 means
-"ready to serve", not merely "process alive". During the first-start download
-it stays unhealthy by design.
-
-## Troubleshooting
-
-| symptom | cause |
-|---|---|
-| `dmalloc: FAILED requesting … (out of memory)` then `HIP … out of memory` | the KV pool did not fit. `HALOGEN_KV_SLOTS` will not fix it (slots share the pool); lower `HALOGEN_KV_POOL_POSITIONS` to 262144, then `HALOGEN_MAX_TOK` to 16384 (~9 % prefill) |
-| starts, answers short prompts, collapses to a few tok/s on a long one, disk busy, process in uninterruptible sleep | short of **file cache**, not memory — lower the pool and/or raise `HALOGEN_HOST_RESERVE_GIB` |
-| long replies that look cut off | read `finish_reason`: `stop` means you have the whole answer, `length` means you ran out of budget (thinking counts) |
-| a request that never returns, then `503 engine_busy` | the pool could not reserve prompt + `max_tokens`; see the admission trap above |
-| both the engine and a co-tenant stall for minutes at full CPU | the pinned-trunk trade; see Memory model |
-| `free`, `MemAvailable`, Grafana all wrong by ~68 GiB | expected — locked weights are counted as reclaimable |
-
-`HALOGEN_VERBOSE=1` ships in the unit, so the fullest startup account is
-already in the log when something goes wrong. `HALOGEN_DMALLOC_LOG` is
-deliberately **not** set: it logs every allocation over 64 MB, which is
-steady-state noise on a shared host — raise it only for a live OOM report.
-The lines worth grepping first:
-
-```bash
-podman logs frank-dragoon 2>&1 | grep -E '^(dmalloc|kv pool):'
-```
-
-Retrieval: `podman logs frank-dragoon` covers the **current** instance only —
-the Quadlet runs `--replace --rm`, so everything before the last recreate is
-gone from the container log. `journalctl --user -u frank-dragoon.service` is
-the durable record. Both are rootless: `sudo podman logs frank-dragoon` looks
-in root's storage and answers "no such container".
-
-## Rollback
-
-The predecessor is intact: the ik payload stays at
-`/home/dank/models/mradermacher-Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`, the
-ik build at `/home/dank/GitPackages/ik_llama.cpp/build`, and its unit, env,
-serve script and the Qwen-Sharp template are archived in the repo under
-`scripts/.AGENTTRASH/*.ik-llama.cpp` and `*.ik-llama-qwen-sharp`. Rolling back
-is a `git revert` of this change set plus a deploy — **but the Quadlet must be
-removed from `~/.config/containers/systemd/` first**, or the restarted root
-unit and the still-installed container both claim port 7770. The port gate in
-the handler now covers exactly that race. See [[frank-dragoon-ik-uncensored]].
-
-The `amd_iommu=off` kernel argument that came with this change is separate and
-reverts on its own — see [[dankclaw-iommu-translation-off]].
-
-## Citations
-
-- [peonist-ai/halogen-flash-server](https://github.com/peonist-ai/halogen-flash-server) — README, `docs/FLAGS.md`, `deploy/entrypoint.sh`, `docker-compose.yml`, CHANGELOG (v0.5.8)
-- [peonist-ai/halogen-qwen3.8-flash-next](https://huggingface.co/peonist-ai/halogen-qwen3.8-flash-next) — weights and file inventory
-- [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) — base model

serverAppConfigs-docs/LLM/index.md

@@ -1,647 +1,89 @@
-# LLM Fleet — Knowledge Base
+# LLM fleet — index
 
-**Dankserver runs the radiance engine (StillDeadcode) on frank-archon**
+The three live endpoint pages are the only pages that carry a shape. Everything
-(digest-pinned `stilldeadcode/radiance` image, engine as entrypoint,
+else here is a technique, a piece of hardware, or a trial that taught a law — and
-driving `podman run` directly, root systemd unit, :7770):
+each of those owns exactly that one thing. The incidents behind the laws are in
-the **publisher's** 4-bit-codebook container of the official
+[[log]].
-`Qwen/Qwen3.8-Flash-Next` (`qwen3.8-next-flash-fp8-iq4r-moe.rad`, 116,322 MiB,
-sha256-verified at fetch — w4nl codebook experts + int8 trunk, PLE n-gram,
-MTP head and vision tower in-container), on BOTH R9700s
-TP=2 (`--tp-wire wht6`), experts in a 44 GiB/rank pinned host pool + VRAM cache
-(explicit `--vram-weights-mib 25600 --vram-kv-mib 5120` = 30,720 of the
-30,969 MiB the engine's own budget table makes claimable), the 47.7 GiB PLE
-table pinned to host RAM, **fp8 KV at 3 × 200,000**, MTP spec decode depth 2,
-froggeric v22.5 template. Rollback: revert the swap commit and re-download the
-in-house Swift1.5 w4dn8 `.rad` byte-exact (purged from disk 2026-10-17; Hub
-`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`); tags
-frank-archon-freetoken-pr587-20261005 and frank-archon-vllm-shapeG-20261003
-(AGENTS.md Runbook Index; `find /home/dank/models -name '*.rad'` on
-Dankserver 2026-10-18 lists only the iq4r file),
-and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2
-(frank-dragoon, port 7770: ghcr.io/peonist-ai/halogen-flash-server @ pinned
-digest serving the NATIVE v2 conversion of the abliterated
-Qwen3.8-Flash-Next, a ~62 GiB v2 .hgn that loads in no other engine,
-MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots over ONE
-401,408-position KV pool — 2 × ctx, raised and DEPLOYED 2026-10-18 — at ctx
-200704, vision ON)** and
-**llama.cpp on the PrismML fork (frank-zealot, port 7771: prism-ml
-Ternary-Bonsai-2-9B at **PQ2_0** — 9B qwen35 hybrid (8 full-attention + 24
-Gated DeltaNet layers + the blk.32 MTP block), TWO slots over ONE
-524,288-position pool (2 × the 262,144 window), MTP present in the file but
-OFF by measurement, **exact f16 K + f16 V** — 32,768 B/token = **16,384 MiB**
-of KV (three 262,144 windows at exact KV do not fit the RTX 3090), vision via
-the publisher's own Q8_0 mmproj, on the RTX 3090 — and the custom
-tensor type 141 + `prism.hadamard.*` metadata mean no other loader can open the
-file)**.
 
-> 2026-08-22: the former `vllmScripts` repository is merged into the
+## Live endpoints
-> serverAppConfigs repository. This knowledge base moved from the old repo's
-> `serverAppConfigs-docs/` root to `serverAppConfigs-docs/LLM/`. The fleet scripts and units now live in the
-> repo's `scripts/` directory.
 
-## Deployment Topology
+| Page | Owns |
+|---|---|
+| [[frank-archon]] | The one frank-archon page: the live radiance shape and harness contract (read off scripts/frank-archon.env), the Eras… |
+| [[frank-dragoon]] | The one runbook for frank-dragoon, the Strix Halo iGPU endpoint (192.168.0.143:7770): current shape from the env,… |
+| [[frank-zealot]] | The one frank-zealot reference: the live shape on Dankclaw's RTX 3090 (:7771) exactly as scripts/frank-zealot.env… |
 
-```mermaid
+## Retired endpoints
-graph LR
-    subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
-        A[2x Radeon AI PRO R9700
-gfx1201, 32 GiB each]
-        B2[radiance StillDeadcode, digest-pinned image
-podman run, frank-archon, port 7770]
-        C[Publisher qwen3.8-next-flash-fp8-iq4r-moe .rad
-2x R9700 TP=2 wht6, experts 44 GiB/rank pinned pool + VRAM,
-PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 2,
-3 x 200000, froggeric v22.5]
-    end
 
-    subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
+| Page | Owns |
-        E2[RTX 3090 (PCIe x4 dock) +
+|---|---|
-gfx1151 iGPU 114 GiB GTT]
+| [[frank-medium-batch-ubatch]] | llama-bench sweep of --batch-size against --ubatch-size on frank-medium (2x Radeon AI PRO R9700, layer split) at the… |
-        E[flash-server container gfx1151, digest-pinned image
+| [[frank-medium-llamacpp-dflash2]] | The former ACTIVE frank-medium deployment since 2026-09-16 (model swapped 2026-09-19) — mainline llama.cpp on… |
-frank-dragoon, port 7770]
+| [[frank-medium-model-swap]] | How to swap the checkpoint behind the frank-medium endpoint (Dankserver :7770, vllm-radiance TP=2) — the streamlining… |
-        F[flash-server v2 native abliterated .hgn + ngram table + vision tower
+| [[frank-medium-radiance-mxfp4]] | ACTIVE frank-medium runbook — vllm-radiance serving hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 (ParoQuant MXFP6 W4A8,… |
-ONE 401408-position KV pool at ctx 200704 = 2 slots x ctx, MTP always-on, image input]
+| [[frank-medium-rocmfpx]] | charlie12345/ROCmFPX llama.cpp serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF as frank-medium on Dankserver… |
-        G[llama.cpp PrismML fork CUDA build
+| [[frank-npu]] | The retired FastFlowLM endpoint on Dankclaw's XDNA2 NPU — what it served, why it died on 2026-09-11 with the… |
-frank-zealot, port 7771]
+| [[frank-redo-2026-08-17]] | This page records the full redo of the inference fleet on 2026-08-17. |
-        H[Ternary-Bonsai-2-9B PQ2_0 GGUF + Q8_0 mmproj
+| [[ikllama-ornith-dankserver]] | ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-zealot on the RTX 3090. Q8_0 model, KV K default f16 / V… |
-2 slots over ONE 524288-position pool at ctx 262144, exact f16 K + f16 V, MTP off, mmproj vision]
+| [[kingjones-q8agent-rocmfpx]] | **Status (historical, 2026-09-13):** was to deploy on Dankclaw. |
-        E2 --> E --> F
+| [[qwen3.8-spec-matrix]] | This page records the speculative-decoding benchmark history and the |
-        E2 --> G --> H
-    end
 
-    CLI[LLM Clients / API] -->|HTTP :7770| B2
+## Pipeline Parallelism (series)
-    CLI -->|HTTP :7770| E
-    CLI -->|HTTP :7771| G
-```
 
-## Servers
+| Page | Owns |
+|---|---|
+| [[Ethernet Bandwidth Math]] | Bytes-per-token math for pipeline vs tensor parallelism, feasibility tables for 1/2.5/10 GbE, and real multi-node… |
+| [[Expected Performance]] | Estimated tokens per second for a 25-40B model split across two nodes over 1/2.5/10 GbE, with real benchmark anchors. |
+| [[llama.cpp RPC]] | llama.cpp distributed inference via ggml-rpc-server and the --rpc flag: architecture, CLI, versions, limitations, and… |
+| [[Mixed-Vendor Experiments]] | Real-world NVIDIA + AMD Strix Halo multi-node experiments: what works (llama.cpp RPC, Vulkan split), what breaks… |
+| [[Qwen3.6 27B Spec Methods]] | Qwen3.6-27B Q8 GGUF variants and llama.cpp speculative decoding (draft-mtp, draft-dflash, ngram-mod): flags, RPC… |
+| [[Qwen3.8 PP Deployment]] | Split Qwen3.8-27B across the Strix Halo iGPU (head) and the RTX |
+| [[Sample Script Set]] | An educational two-node llama.cpp RPC split of Qwen3.6-27B-Q8_K_P over ethernet. For reading only. Do not run or… |
+| [[vLLM Pipeline Parallelism]] | vLLM multi-node pipeline parallelism: flags, Ray/mp backends, interconnect requirements, PP bubble, and PP +… |
 
-| Server                                  | GPU                           | VRAM / Memory        | OS           |
+## Techniques, hardware, trials
-| --------------------------------------- | ----------------------------- | -------------------- | ------------ |
-| [[Dankserver System Specs\|Dankserver]] | 2x AMD Radeon AI PRO R9700 | 2x 32 GB VRAM, 246 GB RAM | Ubuntu 26.04 |
-| [[Dankclaw System Specs\|Dankclaw]]     | AMD Radeon 8060S (Strix Halo) | gfx1151 unified memory | Ubuntu 26.04 |
 
-## Active Deployment
+| Page | Owns |
-
+|---|---|
-- [[frank-dragoon-flash-server-v2-native]] — **ACTIVE since 2026-10-17**
+| [[Dankclaw Improvements]] | This page collects concrete, sourced recommendations to improve Dankclaw |
-  (root systemd unit + podman, port 7770): the digest-pinned
+| [[Dankclaw System Specs]] | Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference. |
-  `ghcr.io/peonist-ai/halogen-flash-server` 0.16.2 engine serving the
+| [[dankclaw-iommu-translation-off]] | Dankclaw boots iommu=pt since 2026-10-18 — identity DMA with the IOMMU armed, which restored the XDNA2 NPU at a… |
-  **native v2 conversion** of the abliterated Qwen3.8-Flash-Next
+| [[Dankserver System Specs]] | Dankserver hardware and OS state since the 2026-09-20 board swap (ASUS ProArt B850-Creator WiFi NEO, 2x AMD Radeon AI… |
-  (`qwen38-flash-next-v2-abliterated.hgn`, 66,687,678,432 B, Hub LFS
+| [[dankserver-frontier-moe-feasibility]] | VERIFIED 2026-09-28 by measurement and source reading. Compares DeepSeek-V4.1-Flash, MiMo-V2.6-Flash-MOPD and… |
-  sha256-verified at fetch) with the REQUIRED n-gram lookup table (51.2 GB,
+| [[dankserver-vram-growth]] | Investigation of the recurring cuBLAS crash on frank-zealot. What grows, what does not, and what to do next. |
-  @peonist 5cc17cea) and the vision tower (897.9 MB, @e053f488)
+| [[dealign-ornith-crack]] | dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with… |
-  **hardlinked** beside it — the card publishes only the main file, and the
+| [[deployment]] | Scoped deployment and update process for the three inference services (frank-archon, frank-dragoon, frank-zealot). |
-  engine's own tools certified the stock companions (`niah` 18/18, `ppl`
+| [[DFlash DSpark Speculative Decoding]] | DFlash and DSpark are the two newest speculative decoding methods. |
-  3.968 with mean KL 0.155 against the Swift1.5 logits on the same corpus).
+| [[Dual-R9700-Cooperation]] | Why frank-medium's two Radeon AI PRO R9700s could not run tensor-parallel on the old B650-E board (PCIe atomic… |
-  `HALOGEN_CTX=200704` over `HALOGEN_KV_POOL_POSITIONS=401408` (**2 × ctx**):
+| [[FastFlowLM NPU Inference]] | FastFlowLM (FLM), the AMD NPU-first LLM runtime for Ryzen AI XDNA2 chips. Fit analysis for the 50 TOPS NPU on… |
-  **one pool behind two slots**, sized so TWO maximum-length requests coexist —
+| [[FreeToken Builds]] | Building FreeToken (the official franksdank fork of FreeTok Tokenworks' freetoken vLLM-MTP runtime) on Dankclaw — the… |
-  raised from 200,704 (5.5 GiB) on 2026-10-18 for frank-zealot parity on total
+| [[freetoken-benchmark-sweep]] | KV pool vs decode speed sweep across bs1/bs3/bs4/bs6 on Dankserver with Qwen3.6-35B-A3B-FP8 via FreeToken v0.1.2. |
-  in-flight context and **deployed the same evening**: the boot ledger reserves
+| [[GGUF Compatibility]] | How to answer "will engine X load this GGUF?" from the file's header alone, why a custom tensor type forces the… |
-  `401408 positions (about 11.0 GiB)` with `79.6 GiB in all`
+| [[Heterogeneous Split Triage]] | Triage before building when someone asks to split a model across unequal accelerators on one host (24 GB GDDR6X +… |
-  (`scripts/frank-dragoon.env:70-75`). It bought CONCURRENCY, not a longer
+| [[Hybrid CPU GPU Processing Dankserver]] | Dankserver uses NVIDIA GPU with system RAM for hybrid processing. |
-  window: the per-request ceiling stays 200,704, because 262,144 is the
+| [[ling-3.0-flash-trial]] | `bailingmoe3-support` (commit `db4480bc8`, build b10411) at `/home/dank/llama.cpp-ling`. |
-  native wall and the engine refuses more without `HALOGEN_ROPE_YARN`.
+| [[ling-3.0-flash-vllm-plan]] | This page was the plan for a vLLM-ROCm deployment of `jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native` (INT4 W4A16, ~62… |
-  MTP head always on, sampler 1.0/0.95/20, coupled cap/timeout
+| [[Llama Cpp Multi-Device Splits]] | What llama.cpp can and cannot express across devices, the no-collective layer-split path, split balancing by… |
-  65536/7200, prompt cache ON, vision ON. Measured against the Swift1.5
+| [[Llama Cpp Serving Sizing]] | Slot-pool sizing against a headroom band, per-device VRAM accounting, prompt-cache/state sizing and the QSA… |
-  boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18,
+| [[LLM Bench Windows and Measurement]] | Stopping-and-restoring benchmark windows on the fleet LLM hosts (llama-bench sweeps, HTTP param benches, multi-arm… |
-  PPL +2.3 %, thinking tokens ×1.9 (the finetune's compression is gone).
+| [[LLM Performance Diagnosis]] | Probing why a served LLM is slow — the flat-depth probe, the expert-offload duty-cycle arithmetic, kernel-time… |
-  Rollback: revert the swap commit; the swift15 payload and its fetch script
+| [[Model Swap and Rollback]] | The full model-swap discipline for the LLM fleet — fit preflight, fetch and payload verification, runtime build/flag… |
-  stay on disk
+| [[PCIe Link and Atomics]] | Link probing (idle downshift, path ceilings, CUDA/HIP bandwidth probes), the streaming-vs-compute ceiling arithmetic,… |
-- [[frank-dragoon-flash-server-v2-swift15]] — **RETIRED 2026-10-06 → 10-17
+| [[Quantization by Hardware Architecture]] | This page maps LLM quantization formats to the GPU and CPU architectures that support them. |
-  (rollback target: payload + fetch script on disk)**: the Swift1.5-finetuned
+| [[Quantization Production]] | — |
-  v2 `.hgn` (compacted r2, 66,687,678,464 B) at TWO slots at ctx 262,144 over
+| [[qwen3.8-strix-optimizations]] | This page answers one question: **how do we make Qwen3.8-27B serve faster on |
-  a 524,288-position pool; owns the 400-zone law (the harness chars/4
+| [[RAM CPU Offloading in VLLM]] | vLLM uses CPU RAM and swap space to extend GPU memory capacity. |
-  under-count against the engine ceiling), the MTP-always-on law, and the
+| [[RDNA4 Container Serving]] | Bringing a model larger than VRAM onto Dankserver's 2x R9700 (gfx1201) with the tcclaviger vLLM stack — quant-class… |
-  sampler / prompt-cache / vision contract the native era carries verbatim
+| [[RDNA4 MoE Expert Offload VRAM Budget]] | The expert-offload plan arithmetic on the tcclaviger fork — which terms a deploy can move, the util law, the… |
-- [[frank-dragoon-halogen27b-dflash2]] — **RETIRED 2026-10-05 → 10-06
+| [[Running Qwen3.6 27B FP8 on Dankclaw]] | Qwen3.6-27B is a dense vision-language model from the Qwen3.6 series. |
-  (rollback target; payload + fetch script + image on disk)**: the
+| [[Running Qwen3.6 35B A3B FP8 on Dankserver]] | Qwen3.6-35B-A3B is a Mixture-of-Experts vision-language model from the Qwen3.6 series. |
-  digest-pinned `ghcr.io/peonist-ai/halogen` 0.1.4 engine serving
+| [[Shared Memory Architecture Dankclaw]] | Dankclaw uses AMD Strix Halo APU with unified memory architecture. |
-  `peonist-ai/halogen-qwen3.8-27b` (`qwen3.8-27b-p1w4d-d2.hgn`, 33.4 GiB)
+| [[swift15-campaign-worklog]] | — |
-  — DFlash2 drafter, ONE 200,700 slot, text-only (the engine ignored the
+| [[Tcclaviger Fork Ops]] | Operating the tcclaviger RDNA4 vLLM fork on dankserver (shape G era, rollback target since the 2026-10-05 FreeToken… |
-  vision encoder), censored base. Owns the measured lessons: the prompt
+| [[ubatch-benchmark-large]] | Standardized ubatch sweep for the active kingjones Qwen3.8 ROCmFPX deployment. |
-  cache pin (`HALOGEN_CACHE_MB=16384` — auto was OFF at the 36.9 GB
+| [[vLLM Boot Forensics and Config Interrogation]] | Capturing a vLLM container boot failure's real root cause from journald (the container log is already gone),… |
-  two-entry floor; a 17.3k cold prefill of 27.0 s resumed in 1.7 s warm),
+| [[VLLM Index]] | This index contains all documentation for vLLM deployment on Dankserver and Dankclaw. |
-  the FA tile rejection (`HALOGEN_FA_TQ=256`: prefill −9.5 %, decode
+| [[VLLM Overview]] | vLLM is a fast and easy library for large language model inference and serving. |
-  ±0.3 %), and deep decode at 3.3-3.8 t/s@80k being structural. The
+| [[vLLM Porting and Compile Triage]] | Forward-porting a model architecture to a new vLLM major (additive-only discipline, gap enumeration, shims that… |
-  foreground-`podman run` ledger capture and the deep-role rerouting
+| [[vllm_research_findings]] | vLLM v0.26.0 (latest stable, released July 25, 2026) supports DFlash speculative decoding with the `--spec-method… |
-  (`slow`/`advisor` → frank-archon) carried forward. Superseded by the
+| [[x4-pair-experiment]] | What mainline llama.cpp can and cannot do with a 3090 + Strix Halo combination, measured 2026-09-16 on Dankclaw — the… |
-  MODEL swap: the abliterated v2 .hgn loads only in flash-server, so the
-  engine moved with the model
-- [[frank-zealot-llama-prism-bonsai9-mtp]] — **ACTIVE since 2026-10-18**
-  (root systemd unit, port 7771): the **PrismML fork** build of llama.cpp
-  (vendor **10777**, commit `e3de71522` — the only loader for the payload's
-  custom PQ2_0 tensor type + `prism.hadamard.*` transform, and since this swap
-  also the minimum build that pairs MTP with a quantized V cache) serving
-  prism-ml/Ternary-Bonsai-2-9B-gguf **PQ2_0 with the MTP head IN the file** on
-  the RTX 3090 — **TWO 262,144-token conversations**
-  (`--ctx-size 524288 --parallel 2`, every slot at the window the model was
-  trained at), **unquantized f16 K + f16 V** (32,768 B/token over the 8 full
-  layers = **16,384 MiB** exact; the POOL halved because that rate over the
-  OLD 786,432 pool bills 24,576 MiB against the 24,123 the driver leaves
-  addressable, and the old "26,112 MiB" note was wrong — it doubled the
-  q8_0 figure (17,408 B/token = 13,056 MiB) *including* its 1/32 scale
-  overhead, so f16/q8_0 is **1.882×**, not 2×), the q8_0-era boot settled
-  **17,876 used / 6,249 free**, vision via the publisher's **Q8_0** mmproj
-  (Q8_0 is the publisher's requirement for concurrent image requests, and this
-  runs two slots), **MTP OFF by measurement** — the draft context bills its own
-  f16 KV over the WHOLE pool (4,096 B/position × 524,288 = 2,048 MiB) and at
-  `UBATCH_SIZE=2048` that squeeze halves decode (167.2 → 109.2 t/s);
-  `SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` is the documented 213.5 t/s arm,
-  declined for +17 % prefill and 6.2 GiB of headroom — and exact KV leaves an
-  f16-draft arm no room at all (a quantized draft, `--spec-draft-type-* q8_0`,
-  is the unmeasured door).
-  Measured at the q8_0-era shape: 167.2 t/s single-stream decode, 2,877 t/s
-  prefill at 106,418, 280.2 t/s aggregate across three streams. `--metrics`
-  rides (`METRICS=1`):
-  llama.cpp answers 501 on `/metrics` without it, which is why Prometheus
-  `up{job="frank-zealot"}` read 0 for its whole retained history until
-  2026-10-18. `contextWindow: 262144` is the **widest in the fleet**, so this
-  is the compaction hub. The `--presence-penalty` opt-out law (`${VAR=}`, not
-  `${VAR:=}`) and the `TimeoutStopSec=180` stop-drain law were both earned here.
-- [[frank-zealot-llama-prism-bonsai2-pq20]] — **RETIRED 2026-10-17 → 10-18**
-  (payload + fork build 10754 tree on disk as rollback): the first PrismML-fork
-  era — prism-ml/Ternary-Bonsai-2-**27B**-gguf **PQ2_0**, TWO 200,704-token
-  conversations (`--ctx-size 401408 --parallel 2`), q8_0 K + q8_0 V
-  (34,816 B/token over the 16 full layers = 13,328 MiB; f16 needs 25,088 MiB and
-  dies), settled 22,664 used / 1,461 free, MTP **impossible** (that export
-  carries no head), 76.6 t/s decode, 691.7 t/s prefill at 166k, 109.4 t/s
-  aggregate at two streams. Owns the 34 KiB/token KV law, the f16 OOM wall, the
-  two-boot VRAM pricing, and the record of the 2026-10-18 bench window
-  (production unit stopped 03:38, hand-booted 9B candidate answering the port)
-  that closed with this swap.
-- [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — **RETIRED 2026-10-07 →
-  10-17** (payload + master CUDA build on disk as rollback): llama.cpp master
-  (CUDA sm_86) serving HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive
-  **Q6_K** on the RTX 3090 — TWO 200,704-token conversations
-  (`--ctx-size 401408 --parallel 2`), exact f16 K + f16 V, vision via the
-  publisher's BF16 mmproj. MTP is OFF because the file has no MTP tensors at
-  all. Priced from the MiMo Q6_K boots (same qwen35 geometry, same card, same
-  shape). Supersedes the Gemma4 era (below).
-- [[frank-zealot-llama-gemma4-q4km-200704]] — **RETIRED 2026-10-07** (same
-  day; superseded by qwen35-9b-q6k; payload on disk as rollback): llama.cpp
-  master (CUDA sm_86) serving HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-
-  HauhauCS-Balanced-MTP **Q4_K_M** on the RTX 3090 — ONE 200,704-token
-  conversation, f16 K + q8_0 V, vision via the publisher's mmproj. MTP was
-  OFF (the gemma4-assistant drafter costs ~1.9 GiB: 23,806 used/319 free with
-  it, 21,890/2,235 without).
-- [[frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688]] — **RETIRED
-  2026-10-07** (superseded by gemma4-q4km; payload on disk as rollback);
-  ACTIVE 2026-10-05 (evening) → 2026-10-07; UNCECERSED RCO-IQ3_XXS with
-  MTP back on since
-  2026-10-06 evening; RENAMED from frank-small and reshaped to 2 ×
-  114,688 / ubatch ladder 512 → 1024 → **2048** late 2026-10-06** (root
-  systemd unit, port
-  7771): llama.cpp master
-  (CUDA sm_86) serving ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-
-  GGUF **RCO-IQ3_XXS-mtp** (27B; the MTP head in the file is DRAFTED —
-  draft-mtp depth 3, ngram skipped on the recurrent-rollback law) on the
-  RTX 3090, TWO 114,688-token conversations at Frank's 112k cap (pool
-  229376 / parallel 2 — cut from 131,072 to fund the ubatch rungs;
-  **1,005 MiB free at boot, 763 settled after a vision probe**) with
-  q8_0 K + q8_0 V (FP8 class, 34 KiB/token vs 64 at f16) and
-  vision through the publisher's own mmproj-BF16.gguf. Prefill 1,209
-  t/s @ 3,760; decode ~52–58 t/s recipe (ubatch-invariant; acceptance
-  noise 34–54 %).
-  RCO-IQ3_XXS tier is a per-tensor allocation mix of stock types; the
-  payload is the UNCECERSED Swift 1.5 (23/100 refusals vs Swift 1.5's
-  98/100). Replaces BOTH same-day payloads: the live MiMo Q6_K shape
-  (below) and the Ornith-1.5-9B Q8_0 swap that was committed
-  (77decb5..380aafa) but never deployed
-- [[frank-zealot-llama-mimo-q6k-2x200704]] — **RETIRED 2026-10-05 (evening;
-  rollback target; payload on disk)**: llama.cpp master (CUDA sm_86)
-  serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, TWO slots of
-  200,704 with FULLY UNQUANTIZED KV (f16 K + f16 V) and the mmproj for
-  vision — 94.4 tok/s decode (the rung costs a measured -10 % vs Q5_K_M:
-  weight-bandwidth-bound), batch 4096 / ubatch 2048 since 2026-10-05
-  (+4.6 % prefill, measured), 2,279 MiB of the card left free. This was
-  the LIVE payload when Swift cut over. The ubatch A/B table and the
-  probe-size trap (>=256 px images or the model says "video") live inside
-- [[frank-zealot-llama-mimo-q5km]] — **RETIRED 2026-10-02 → 10-03 (rollback
-  target; payload on disk; owns the measured boot-pair table)**: the same
-  runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower —
-  104.8 tok/s, 3,571 MiB true free
-- [[frank-archon-radiance-qwen38-iq4r]] — **ACTIVE since 2026-10-15**:
-  the engine author's published container of the OFFICIAL (base-alignment)
-  Qwen3.8-Flash-Next — `qwen3.8-next-flash-fp8-iq4r-moe.rad`, 116,322 MiB,
-  Hub-sha256-verified at fetch: 4-bit codes into a 16-level non-uniform
-  codebook after a Walsh–Hadamard rotation (GPTQ + 3 coordinate-descent
-  sweeps, 10M-token calibration), ten bf16-protected experts per layer,
-  int8 trunk/lm_head, fp8 HC + PLE, MTP head, bf16 vision tower; card KL
-  0.0789 all-tokens / top-1 92.85 % vs bf16. **MODEL_FILE is the only line
-  that moved** — the 2026-10-13 shape (25600/5120 pair, 44928 pool, fp8 KV
-  3 × 200,000, MTP 2, wht6, MNBT 2048, froggeric v22.5) carries verbatim.
-  Rollback: revert the swap commit (w4dn8 `.rad` stays on disk). The
-  fp8-native `.rad` and the raw Swift1.5 BF16 source were DELETED from
-  dankserver 2026-10-15 (Frank's one-time rm, volume for this payload;
-  rebuild via the fetch + build scripts named in the fp8-native runbook)
-- [[frank-archon-radiance-swift15-w4dn8]] — **RETIRED 2026-10-08 → 10-15
-  (superseded by the publisher iq4r swap; the `.rad` was PURGED from disk
-  2026-10-17 — rollback = revert + byte-exact re-download of
-  `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`)**:
-  Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 `.rad` on radiance
-  1.0.13 (recipe + 10M-token calibration + KLD gate vs the BF16 container);
-  the FP8-native payload was the second rollback until it too was deleted
-  2026-10-15. **2026-10-13: fp8 KV, and the
-  pair re-cut twice** — `--kv-cache-dtype fp8` first at
-  `--max-num-seqs 4`/6912 MiB pool (843,566 tokens, slab 22208), then **3 slots
-  with the pool cut to the contract (5120 MiB, 608,027 tokens) and the slab
-  raised 22208 → 24000 → 25600**, the last 1,600 MiB claimed out of the card's
-  settling slack: the engine's ceiling is **30,969 MiB**, not the 29,120 my
-  1 GiB reserve invented. fp8 is what funds the slots — the pool must back
-  `seqs × window`, and at bf16 four slots cost 11,679 MiB (slab 17,441, −33 %).
-  Measured on the slab: A-cold 43.5k prefill 22208 → 3,397, 24000 → 3,290,
-  **25600 → 3,584 t/s**; 3-way 197k aggregate 3,051 → **3,589 t/s**. A shipped
-  prediction that the slab was a threshold below 26 GiB died on its own
-  measurement — it is a slope, and 26,071 (4,379 t/s) is the cliff. The bf16
-  arm (2 × 200,000, 462,752 tokens, prefill −29 %, acceptance 0.661 → 0.740)
-  and the 4-slot fp8 shape are the documented rollbacks.
-  **2026-10-13 also: the pinned-ledger preflight now raises `ttm.pages_limit`
-  itself** — the 2026-10-12 power failure
-  reset the knob to its default (125,988 MiB) and the endpoint refused to boot
-  6,919 times in ~20 h; `scripts/ttm-preflight.sh` is the repair.
-- [[frank-archon-radiance-qwen38-native-fp8]] — **RETIRED 2026-10-08 (the
-  w4dn8 swap replaced it; its `.rad` was DELETED from disk 2026-10-15 —
-  rebuild = `scripts/download-frank-archon.sh` + `scripts/build-frank-archon-rad.sh`,
-  `scripts/frank-archon.env:59-62`)**
-  (root systemd unit, port 7770): the StillDeadcode **radiance** engine
-  (C++/HIP, libr4d gfx1201 kernels, digest-pinned 287 MiB image, engine as
-  the image entrypoint) serving the OFFICIAL `Qwen/Qwen3.8-Flash-Next-FP8`
-  checkpoint at NATIVE precision — converted no-recipe to
-  `qwen4exp-fp8-native.rad` (~161 GiB) after dropping `quantization_config`
-  in a staging dir (the plugin descriptor-selection law: nothing claims
-  `(qwen4exp, "fp8")`, the shipped plugin claims `(qwen4exp, "")`;
-  per-weight encodings are data, not the descriptor) — on BOTH R9700s
-  **TP=2** (`--tp-wire wht6` fused all-reduce), experts in a 40 GiB/rank
-  pinned host pool + VRAM cache (ratio 0.82 = the measured 99 %-hit point),
-  the 47.7 GiB PLE n-gram table PINNED to host RAM (`--ngram-placement
-  ram`), fp8 KV 7.32 KiB/token, MTP spec decode depth 3, 2 × 200,000 first
-  boot, froggeric v22.5 template, vision + tools + `reasoning_content` auto,
-  `/metrics` scraped. Rollback: git tag
-  `frank-archon-freetoken-pr587-20261005` (FreeToken image + orcarouter
-  payload on disk); older rollback `frank-archon-vllm-shapeG-20261003`
-- [[frank-archon-freetoken-pr587]] — **RETIRED 2026-10-05 (same day;
-  rollback target)**: FreeToken at PR #587 head `77a7201` (branch
-  rocm714_torch214) built on the host into the local image
-  `frank-archon-freetoken:pr587-77a7201` (Dockerfile: digest-pinned
-  rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev-symlink law,
-  curated --no-deps install per docs/install_amd.md), serving the SAME
-  orcarouter FP8 payload on ONE R9700 — experts in pinned host RAM
-  (~120 GiB + 47.7 GiB PLE), GPU LRU at 0.10, KV pool 299,968 tokens
-  (multiple-of-64 law), 2 slots, parsers AUTO (thinking →
-  `reasoning_content`). Measured: boot ~2.3 min, decode 17.1 tok/s
-  single-stream, vision "Red" probe passes, native tool_calls; TP=2 does
-  not boot on ROCm (pynccl JIT) — second card idle. Superseded by the
-  radiance swap for TP=2. Disk state 2026-10-18: the image is still in
-  dankserver's rootless storage (`localhost/frank-primary-freetoken:pr587-77a7201`,
-  29.7 GB — the pre-rename name); the orcarouter FP8 payload is NO LONGER on
-  disk (no `*orcarouter*` path under /home/dank, `find` 2026-10-18)
-- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **RETIRED 2026-10-05
-  (rollback target; the vLLM trio + env lore live in the tag)**: the
-  tcclaviger RDNA4
-  vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both
-  R9700s, TP=2 with automatic expert offload — `--max-model-len 200704` x 3
-  slots, fp8 KV, pool 3,150,000,000 B/rank (measured 398,658 tokens = 1.99x),
-  MNBT 1024, gather 32x64, no spec decode, prefetch off, plus
-  `--expert-precise-memory` (arena 17.22 GiB / 7,522 slots), GPU_MAX_HW_QUEUES=2
-  and a mounted compile cache. Why `--max-num-seqs`
-  must stay 3 (SEQS 2 cannot compile this model), why the arena's levers are
-  `max_model_len` and the overhead term rather than the KV pool, and the measured envelope
-- [[frank-archon-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
-  failed GLM experiment; frank-archon runs Qwen FP8 again, see the intro):
-  the tcclaviger RDNA4 vLLM fork
-  (digest-pinned `tcclaviger/vllm@sha256:da4029666…`, plain `podman run`,
-  no shim) serving Intel/GLM-5.3-Flash-W4A16-AutoRound — AutoRound symmetric
-  int4 group 128, ~170 GiB, 34 shards + 2 extras, 113,074 tensors — on BOTH
-  R9700s, TP=2, with the MoE experts offloaded to pinned host RAM (12,096
-  experts/rank, 73.64 GiB) behind a 14.51 GiB / 2,383-slot VRAM LRU arena; the
-  MoE kernel is libr4d's grouped W4A16 GEMM (`R4dW4a16MoEMethod`, reached
-  because the checkpoint's `quant_method: auto-round` is overridden to `inc` —
-  a compressed-tensors W4A16 checkpoint of the same model has NO offload recipe
-  and refuses to serve). 131,072 ctx x 2 seqs, bf16 KV, util 0.98, prefix
-  caching, vision and `glm47` tool/reasoning parsers, MTP OFF and prefetch OFF
-  by measurement. **Measured decode 9.9-10.0 tok/s, prefill 918 tok/s — the
-  35 tok/s target is NOT met, bounded by the fork's expert-gather read path
-  (~8.2 GB/s effective vs a 22.74 GB/s link) at a 0.53-0.66 cache hit rate.**
-  Rollback: commit `2339fae` (the qwen FP8 + MTP4 shape of 2026-09-29 → 09-30)
-  [[frank-archon-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
-  back; the R9V state dirs are on disk (`/home/dank/r9v-state`,
-  `~/.local/state/r9v`) but the payload GGUFs are not — `/home/dank/models`
-  holds only the iq4r `.rad` as of 2026-10-18; git tag
-  frank-archon-r9v-q8-20260929 is the last pre-cutover commit):
- [[frank-archon-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
-  (same-day swap to the Q8_0 payload; state + payloads no longer on disk
-  2026-10-18, only the R9V state dirs remain): the R9V radiance-vLLM fork serving
-  Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 shards) +
-  blockfp8 MTP4 (depth 4) + F16 vision on BOTH R9700s — TP=2 tiered
-  experts (UVA pinned host copy, 62 hot + 160-slot LRU on rank 0, 428
-  static on rank 1), PLE n-gram on ssd-mmap, CED on, 196608 ctx. Cut over
-  from llama.cpp the same day; rollback target
-  [[frank-archon-llamacpp-qwen38-flash-next-q8]]
-- [[frank-archon-llamacpp-qwen38-flash-next-q8]] — RETIRED 2026-09-27
-  (rollback target; the llama.cpp b11030 endpoint of 2026-09-26):
-  unsloth llama.cpp (b11030-mix prebuilt, ROCm gfx120X, bundled runtime)
-  serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 across both R9700s —
-  routed experts and the 51B-param n-gram table computed in host RAM
-  (`-ot exps=CPU,per_layer_token_embd.weight=CPU`), the shared Q8_0 MTP head
-  (depth 2) chained behind ngram-mod, BF16 vision, one full 262,144-token
-  conversation, `reasoning_content`, the placement law, the cutover and
-  rollback
-- [[frank-archon-lucebox-ds4]] — RETIRED 2026-09-26 (same-day swap,
-  rollback target; active for hours): the Lucebox DS4.1 engine
-  (localhost/lucebox-rocm:gfx1201) serving
-  Lucebox/DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB) with streamed experts —
-  the fleet's only non-llama.cpp runtime detour
-- [[dankserver-frontier-moe-feasibility]] — VERIFIED 2026-09-28: which of
-  DeepSeek-V4.1-Flash (552B), MiMo-V2.6-Flash-MOPD (309B) and GLM-5.3-Flash
-  (320B) can run on the 2x R9700 box, on what runtime, and why V4.1 does not
-  fit (475 GiB payload, 189 GiB of it Engram; decode is PCIe-expert-stream
-  bound, ~23 tok/s ceiling at today's residency). Includes the Affinity
-  V4-Flash-to-V4.1 port assessment — reusable expert tiering, four
-  architecture gaps, one of them a silent-wrongness trap on the Engram key
-- [[dankserver-qwen38-fp8-expert-offload-tuning]] — VERIFIED 2026-09-29:
-  the flag-level decode-tuning campaign on the ad hoc qwen-fp8 container
-  (tcclaviger RDNA4 fork, orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8,
-  TP=2 across both R9700s). **>35 tok/s MET: 37.44-40.57 tok/s with
-  `--expert-prefetch` DELETED** — telemetry measured the prefetcher at
-  36% precision, wasting ~half the gather traffic (the LRU itself runs at
-  82.45% hit). Prefetch-on incumbent 26.8 survived every other challenger
-  (geometry, spec depth, TunableOp-corrupts, batch knobs, concurrency,
-  ngram-fork-bug); the residual gap to ~60 tok/s is the fork's gather
-  read path (6.5-9.5 GB/s effective vs 22.7 at the same slab size)
-- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26 (replaced by
-  [[frank-archon-lucebox-ds4]], then
-  [[frank-archon-llamacpp-qwen38-flash-next-q8]]): vllm-radiance 0.9.3
-  serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark
-  W4A8) with the z-lab DFlash2-FP8 drafter as ONE tensor-parallel process
-  (TP=2) behind :7770, on unquantized bf16 KV (`KV_DTYPE=auto`, cut
-  2026-09-21). The FP8 checkpoint is the on-disk rollback. The three-arm
-  battery that chose TP=2, the auto-sized KV pool, and the verified
-  capability battery (tools, reasoning)
-- [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium:
-  the streamlining toolchain (hf-manifest.py, hf-fetch.sh,
-  swap-frank-medium.sh), the quantization-aware entrypoint contract, the
-  quadlet edit, the deploy, the capability battery, and the rollback path.
-  The 2026-09-20 MXFP4 → FP8 swap is the worked example
-- [[frank-medium-llamacpp-dflash2]] — RETIRED 2026-09-16 → 2026-09-20: mainline
-  llama.cpp serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the
-  DFlash2 Q4_K_M drafter as ONE layer-split process (32:33 over two
-  163,840-token slots); superseded by the radiance TP=2 cutover, and the
-  rollback target. Keeps the batch/ubatch optimum curve and the per-layer split
-  arithmetic
-- [[Dual-R9700-Cooperation]] — whether the two R9700s can serve ONE model: the
-  RCCL/PCIe-atomics wall on the old TUF GAMING B650-E, the 2026-09-20 ProArt
-  B850-Creator x8/x8 swap that cleared it, and the measured three-way verdict
-  (radiance TP=2 95.8 t/s decode vs layer split 45.7 vs llama tensor mode 30.5)
-  that put TP=2 in production
-- [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era,
-  2026-09-15): charlie12345/ROCmFPX serving
-  kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the
-  runtime it replaced
-- [[freetoken-frank-small]] — RETIRED 2026-09-15, when the RTX 3090 moved to
-  frank-dragoon: the FreeToken v0.1.2 frank-zealot runbook (batsclamp
-  Huihui-Qwen3.6-35B-A3B-abliterated-FP8), kept as the retirement record. The
-  NAME was reused on 2026-09-16 by a different service on Dankclaw — see
-  [[frank-zealot-qwen38-gsq-rco]]
-- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep
-  (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
-- [[deployment]] — deploy and update process for both servers
-- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
-- [[frank-dragoon-rocmfpx-q6agent-uncensored-mtp]] — **RETIRED 2026-10-03 →
-  10-05 (rollback target; payload, fork build and build script on disk)**:
-  the pinned ROCmFPX fork serving the uncensored protected-heads
-  Q6_0_ROCMFPX_AGENT rung with the repo's MTP-Q4_0 head drafting 4 tokens,
-  one 262,144 window, f16 KV; measured 27.6-28.1 t/s prose decode at 86.8 %
-  acceptance. Replaced by the halogen DFlash2 endpoint; the ngram-mod-onto-MTP
-  rejection (-15 %, recurrent-state rollback) stays a live lesson for this
-  architecture.
-- [[frank-dragoon-rocmfpx-q8agent-mtp]] — **RETIRED 2026-10-02 → 10-03 (the
-  rollback target; payload still on disk)**: the same pinned ROCmFPX fork
-  serving the ALIGNED `Q8_0_ROCMFPX_AGENT` (ftype 115, 17.0 t/s, MTP
-  acceptance 0.462) — the MTP-vs-none +123 % A/B, the trap catalogue
-- [[frank-dragoon-gufo-qwen38-27b]] — **RETIRED 2026-10-02 (history; the
-  payload dirs were removed from disk 2026-10-03, so this rollback is a
-  re-download)** (root systemd unit, port 7770): gufo
-  (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from
-  source on Dankclaw, serving unsloth/Qwen3.8-27B-GGUF **UD-Q4_K_XL**
-  with the z-lab DFlash2 Q4_K_M drafter — ONE session at the full native
-  262,144 window, unquantized f16 KV. Rolled back by Frank after one day
-  of [[frank-dragoon-vllm-mimo-v26-bf16]]. The two from-source build
-  workarounds, the client-contract delta, the measured prefill/decode
-  trade against llama.cpp
-- [[frank-dragoon-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (one day;
-  rollback by Frank after the autoresearch loop): the digest-pinned
-  kyuz0/vllm-therock-gfx1151 image serving
-  XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16, 4 x 262144 ctx,
-  CROSS-MODEL MTP from Qwen/Qwen3.5-9B — kept for the measured record:
-  battery decode 6.19 t/s (MTP-2) vs 9.01 t/s (no draft — spec NET-
-  NEGATIVE on real chat+reasoning traffic), the roof math, the greedy-text
-  numerics trap, and the death of every ≥6-bit vLLM quant path on gfx1151
-  (quark int8 rejected, FP8 kernels CDNA-only, torchao unwired)
-- [[frank-dragoon-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
-  [[frank-dragoon-gufo-qwen38-27b]]; rollback target): stock llama.cpp
-  (unsloth b11030, ROCm gfx1151) serving
-  unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2
-  Q4_K_M drafter (depth 4) on the gfx1151 iGPU — one full 262,144-token
-  conversation, q8_0 KV (the build lacks fp8), text-only, Qwen-Sharp
-  v22.4.0 template, the 114 GiB GTT memory model, client contract,
-  troubleshooting
-- [[frank-dragoon-kairic-edge-qwen38-27b]] — RETIRED 2026-09-21 (same-day
-  swap, replaced by [[frank-dragoon-unsloth-qwen38-27b-q8xl]]; rollback
-  target): the Kairic Edge runtime (ciru-ai/ROCmFPX v1.2, built from
-  source) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (IU4 W4A4, 27B dense)
-  on the gfx1151 iGPU — one full 262,144-token conversation, MTP depth 4 +
-  n-gram drafting, the three PromptForge sidecars, compatibility mode, the
-  114 GiB GTT memory model, client contract, troubleshooting
-- [[frank-dragoon-unsloth-qwen38-flash-next]] — RETIRED 2026-09-21 (same-day
-  swap, replaced by [[frank-dragoon-kairic-edge-qwen38-27b]]; rollback target):
-  the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving
-  unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (125B MoE) on the gfx1151 iGPU —
-  one 200,000-token conversation, MTP depth 2 via the explicit -md shared-Q8_0
-  head, the 114 GiB GTT memory model, client contract, troubleshooting
-- [[frank-dragoon-ciru-orca]] — RETIRED 2026-09-21 (replaced by
-  [[frank-dragoon-unsloth-qwen38-flash-next]]; rollback target): the CIRU
-  v4.4.1 runtime (private host build, QSA hotfix) serving
-  jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 iGPU — one
-  262,144-token conversation, MTP depth 4, 4096 MiB PLE cache, runtime-root
-  gate, client contract, troubleshooting (2026-09-19 → 09-21)
-- [[halogen-flash-server]] — RETIRED 2026-09-19 (replaced by
-  [[frank-dragoon-ciru-orca]]; rollback target): peonist-ai halogen-flash-server
-  0.5.8 serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b
-  .hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one
-  400,384-position pool, deployed flags, memory model, client contract,
-  troubleshooting (2026-09-16 → 09-19)
-- [[frank-zealot-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02** (the same
-  rung returned ACTIVE 2026-10-03 at the 2 x 200,704 f16-KV shape —
-  [[frank-zealot-llama-mimo-q6k-2x200704]]): llama.cpp (master, CUDA sm_86
-  build) serving
-  bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090 — TWO
-  native 262,144-token sessions with ASYMMETRIC KV (K f16 unquantized + V
-  q8_0, which forces flash-attn), ~20.9/24.6 GiB measured, text-only (the
-  GGUF has no vision), reasoning_content, no spec decode; payload still at
-  `/home/dank/models/MiMo-Q6K/`
-- [[frank-zealot-sglang-mimo-v26]] — RETIRED 2026-10-01 (hours; replaced
-  by [[frank-zealot-llama-mimo-q6k]] for the asymmetric KV SGLang cannot
-  express; venv still on disk, INT8 payload staged to
-  `~dank/.AGENTTRASH/20261002-retired-llm-payloads/`): SGLang in a
-  self-contained python3.12 venv serving leteschlead MiMo-V2.6 INT8
-  W8A16 AutoRound, 2 x 262144 ctx, fp8 KV, in-checkpoint vision,
-  83.9 tok/s decode
-- [[frank-zealot-gemma4-12b]] — RETIRED 2026-09-17 → 2026-10-01 (replaced by
-  [[frank-zealot-sglang-mimo-v26]] → the llama.cpp MiMo shape; payload staged
-  to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/`):
-  mainline llama.cpp serving
-  HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
-  3090, with the separate MTP assistant sidecar chained behind ngram-mod,
-  q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB.
-- [[frank-zealot-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
-  rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
-  Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-
-  token pool, 2 concurrent full-context sessions, native text+image+audio.
-- [[frank-zealot-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): the
-  prior frank-zealot runbook (2026-09-16 → 09-17), ISTA-DASLab
-  Qwen3.8-27B-GSQ-RCO IQ3_S with the model's own in-file MTP head and
-  f16 KV at 131k
-- [[x4-pair-experiment]] — MEASURED 2026-09-16: what mainline llama.cpp can and
-  cannot do with the 3090 + Strix Halo pair, over their only link (the 3090's
-  Gen4 x4). The tensor inventory that decides any split, the single flag (`-ot`)
-  that kills per-tensor placement across devices, the two-device layer splits
-  that work (+7 % decode at 64k, cross-vendor included), the harness-parsing trap
-  that invalidated the first round, and the iGPU-only result that matches
-  halogen's deployed decode rate — at a memory cost the co-tenants cannot pay
-- [[frank-dragoon-ik-uncensored]] — RETIRED 2026-09-16 (rollback target): the
-  ik_llama.cpp runbook (root systemd unit, 2026-09-15 → 09-16) serving
-  mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090
-  (dense/attention/KV) + LPDDR (experts/PLE) hybrid; its measured envelope and
-  decode root-cause chain are the record of why the fleet went back to halogen
-- [[dankclaw-iommu-translation-off]] — Dankclaw's IOMMU boot line: **`iommu=pt`**
-  since 2026-10-18, measured on this host at no prefill cost (+2.0 %, 1,616 t/s
-  cold), which restored the XDNA2 NPU; the `amd_iommu=off` era (2026-09-11) it
-  replaced ran on a 13-16 % figure measured upstream only; includes the AMD
-  PerfOpt research, what was deliberately not copied from the reference machine,
-  and the revert path via `scripts/set-dankclaw-bootline.sh`
-- [[kingjones-strix-rocmfp4]] — PRIOR frank-dragoon runbook and rollback
-  target: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params
-  (56B MoE + 51.2B PLE), single 113.47 GiB GGUF, on the card's pinned
-  kingjones30/ROCmFPX fork (2026-08-30 → 2026-09-11)
-- [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results
-- [[frank-medium-batch-ubatch]] — batch/ubatch sweep on the 2x R9700 layer split
-  (2026-09-16): the ubatch optimum at 512, why the logical batch is free, and
-  the split rebalance that followed it
-- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
-- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model
-  on Dankclaw (retired)
-- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on
-  Dankserver (frank-zealot, retired 2026-09-02)
-- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for
-  frank-zealot (2026-08-26): q8_0 + Hadamard KV cache,
-  split-mode none (single-GPU), mandatory FA_ALL_QUANTS build flags
-- [[kingjones-strix-mtp]] — the former Q6 AGENT runbook (restored 2026-09-01
-  after Q8_0 failure, then superseded by the STRIX deployment on 2026-08-27)
-- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime
-  on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
-- [[kairic-edge]] — the retired Kairic Edge IU4 experiment
-  (deployed and reverted 2026-08-22)
-- [[Dankserver System Specs]] — the 2x R9700 box (frank-archon)
-- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-dragoon on
-  the flash-server v2 runtime since 2026-10-06, frank-zealot on the RTX 3090)
-- [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint
-  on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as
-  the retirement record — the IOMMU/SVA err=-19 root cause and where
-  every dependent route moved
-
-## CIRU Reference (Dankclaw)
-
-- [[ciru-activefpx-promptforge]] — the RETIRED v3 model, ROCm 7.15 toolchain,
-  and build history (on Dankclaw; the gfx1151 slot now serves frank-dragoon's
-  flash-server v2-native payload — see [[frank-dragoon-flash-server-v2-native]])
-- [[kairic-edge]] — Kairic Edge IU4 runtime: build runbook, binary
-  identity, and experiment summary (retired 2026-08-22)
-
-## DFlash2 Reference (archived)
-
-- [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment
-  (2026-08-21)
-
-## vLLM Reference (historical)
-
-- [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
-- [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
-- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware
-  notes and sizing (this architecture family is what the retired frank-zealot
-  served in the Huihui abliterated FP8 variant — see
-  [[freetoken-frank-small]]; the Ornith-1.5-35B-A3B CRACK checkpoint
-  that preceded it is archived at [[dealign-ornith-crack]])
-
-## Memory Management
-
-- [[RAM CPU Offloading in VLLM]] — KV cache swap between GPU VRAM and system RAM
-- [[Hybrid CPU GPU Processing Dankserver]] — NVIDIA GPU compute with system RAM offload
-- [[Shared Memory Architecture Dankclaw]] — AMD unified memory and ROCm configuration
-
-## Retired Topology (2026-08-17 redo)
-
-- [[Qwen3.8 PP Deployment]] — the retired pipeline-parallel head + worker
-- [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history (superseded by FastMTP)
-- [[FastFlowLM NPU Inference]] — the retired FastFlowLM NPU endpoint
-- [[frank-dragoon-benchmark]] — backend/spec benchmark behind the old head
-- [[dankserver-vram-growth]] — VRAM/RAM growth behind the retired cuBLAS crash
-- [[DFlash DSpark Speculative Decoding]] — retired DFlash/DSpark drafting
-- [[Quantization by Hardware Architecture]] — which quant format fits which GPU/CPU
-
-## Pipeline Parallelism (archive)
-
-- [[Pipeline Parallelism/index]] — index of multi-node inference research. The PP
-  pair is retired. These pages are research history.
-
-## Runtime Research (llama.cpp family)
-
-- [[Llama Cpp Runtimes]] — mainline vs charlie12345/ROCmFPX vs ik_llama.cpp: positioning, quantization, server fit
-- [[Llama Cpp Code Diff]] — code-level quantization diff of the three runtimes + server recommendations
-- [[SGLang]] — the structured-generation serving engine; when it fits Dankserver (2x R9700) vs Dankclaw (Strix Halo)
-
-## Cross-Cutting References (skill distillation, 2026-10-05)
-
-One polished reference page per skill-cluster, distilled from the managed
-skill library into the vault (see [log](../log.md) for the wave plan):
-
-- [[Quantization Production]] — how we self-make quants per card
-  (imatrix/K_P ladders, radiance recipes, REAP/Heretic/abliteration
-  prior art), and what each runtime decodes natively
-- [[GGUF Compatibility]] — what a GGUF's tensor histogram and header demand
-  of a loader; how the runtime is chosen by the file
-- [[Llama Cpp Serving Sizing]] — slot/pool/KV sizing laws, boot-shape
-  laddering, llama-bench discipline
-- [[Llama Cpp Multi-Device Splits]] — layer vs tensor splits, `-ts`/`-ot`
-  laws, drafter placement
-- [[PCIe Link and Atomics]] — the link walls, atomics routing, and the
-  heterogeneous-pair verdicts
-- [[RDNA4 Container Serving]] — bring-up recipes for RDNA4 containers
-- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan law, every arena
-  lever, and why free VRAM is not arena
-- [[vLLM Boot Forensics and Config Interrogation]] — reading a boot that
-  died, and interrogating a running fork's config
-- [[vLLM Porting and Compile Triage]] — porting checkpoints across vLLM
-  versions/forks; torch.compile/dynamo breakage triage
-- [[LLM Bench Windows and Measurement]] — window discipline, honest A/B,
-  the window script laws
-- [[LLM Performance Diagnosis]] — slow prefill/decode triage on live
-  endpoints, IBS attribution, prefix-cache routing
-- [[Model Swap and Rollback]] — the swap procedure, payload preflight,
-  research verification, second-endpoint bring-up
-- [[Tcclaviger Fork Ops]] — the fork's sweep harness, measured knob
-  verdicts, large-MoE recipe, compile-cache persistence
-- [[Heterogeneous Split Triage]] — triage for unequal-accelerator split
-  proposals (the no-go verdict format)
-- [[FreeToken Builds]] — the CUDA and ROCm build paths with their
-  dependency traps
-
-These sit alongside the operational pages under
-`../Operations/index.md` and `../Agent/index.md`.
-
-## Related
-
-- [Project knowledge base](../index.md) — the Podman app stacks on the same
-  two hosts. OpenWebUI (Dankclaw) consumes the on-host `frank-dragoon`
-  endpoint at `:7770`. The Gitea instance on Dankclaw hosts this repository.
-- [Deployment Flow](../Architecture/Deployment%20Flow.md) — the service-scoped
-  git-based deploy with inference isolation

serverAppConfigs-docs/LLM/kairic-edge.md

@@ -1,327 +0,0 @@
----
-type: reference
-title: Kairic Edge IU4 Runtime on Dankclaw
-description: jcbtc/Qwen3.8-27B-IU4-Kairic-Edge served by the Kairic Edge CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Build runbook, binary identity, serving profile, and rollback.
-tags:
-  - llama.cpp
-  - rocm
-  - dankclaw
-  - gfx1151
-  - kairic
-  - iu4
-  - promptforge
-timestamp: 2026-08-22
----
-
-# Kairic Edge IU4 Runtime on Dankclaw
-
-> **Retired 2026-08-22.** This experiment was deployed, benchmarks were
-> collected, and the service reverted to v3 the same day (operator
-> decision). `frank-dragoon` serves v3 again. This page is kept as
-> the retired experiment's build runbook and reference.
-
-`frank-dragoon` (port 52625) briefly served
-`jcbtc/Qwen3.8-27B-IU4-Kairic-Edge` through the Kairic Edge CIRU ROCmFPX
-llama.cpp runtime. The runtime routes qualified prompt and multi-token
-verification shapes through the RDNA 3.5 native
-`V_WMMA_I32_16X16X16_IU4` matrix instruction. It was reverted to
-[[ciru-activefpx-promptforge]] (v3) on 2026-08-22.
-
-Performance summary of the experiment: the IU4 lane accelerated prefill
-1.5-2.3x over v3 at all depths, but prose decode was 15-26% slower
-(inherent to the release: the IU4 lane does not accelerate M1 target
-decode). Reasoning-off fast-greedy mode was ~8% faster per token than
-production chat, and roughly 2x faster per turn from halved output volume.
-The v3-era experience, including the 400 errors from the argmax fast path
-rejecting non-greedy requests, motivated the revert.
-
-## Identity
-
-The release pins every input. Verify these values before trusting a build.
-
-| Input | Pinned value |
-|-------|--------------|
-| Runtime repo | ciru-ai/ROCmFPX tag `kairic-edge-qwen38-27b-v1`, commit `e97b324` |
-| Composable Kernel | ROCm/composable_kernel `fdf4bb7fcc984811cef48ce817d89aac064b984a` |
-| CK release patch | `patches/composable-kernel-gfx1151-iu4.patch`, SHA-256 `ea57bca3793d35d884fc156b6c0bc33d8e672e3bb49f66ff67338503029d6af5` |
-| Certified toolchain | TheRock 7.15.0a20260718, AMD clang 23.0.0, GCC 13.3.0, CMake 4.4.2, Ninja 1.13.0 |
-| Local toolchain | TheRock 7.15 SDK (ling3-ciru venv), amdclang++ 23.0.0git, GCC 15.2.0, CMake 4.2.3, Ninja 1.13.2 |
-| Frozen certified binary | `llama-server` SHA-256 `0ece51fa4489e58b7c6942ec2aa59baadca897d8cf87f8e1dc4dc0b10a6efc49` |
-| Local binary | `/home/dank/ciru-qwen38/ROCmFPX-kairic/build-kairic/bin/llama-server`, SHA-256 `d7625a0c14ac4158fd793c50ac0cefd8cdcbf76b5aca5cddafa09a353dc8f048` |
-
-The guide states that a portable rebuild does not match the frozen
-certified binary byte for byte. The local toolchain differs from the
-certified one. The local binary reproduces byte-identical across local
-rebuilds (see [Build reproducibility](#build-reproducibility)).
-
-## Model artifacts
-
-Download all four files into one directory. The three `.pfs` sidecars are
-accelerator-specific prepacked weight views. The runtime requires all of
-them.
-
-| File | Size | SHA-256 |
-|------|------|---------|
-| `Qwen3.8-27B-IU4-Kairic-Edge.gguf` | 15.48 GiB | `360caf7381907c3eca7ac0afd1228efc016af747f3f38637fb1c7f94daabac2a` |
-| `Qwen3.8-27B-Kairic-IU4-FFN.pfs` | 7.99 GiB | `adcbb90a7b429a30a2a39043366d68320d72e8b4816a0f498e882b2f80a2ba2b` |
-| `Qwen3.8-27B-Kairic-IU4-GDN.pfs` | 1.88 GiB | `82f931316f1c895da104915dec4697163808d06f0e6b2dc027cee7aa3afc0f0e` |
-| `Qwen3.8-27B-Kairic-IU4-GDN-Output.pfs` | 0.70 GiB | `3b07e7b176559e4402924ba0c368532fa6f02118a33c71e70974c809bf6208a3` |
-
-Total payload: 26.05 GiB. Local directory:
-`/home/dank/models/Kairic-Qwen3.8-27B-IU4-Edge/`. All four SHA-256 values
-verified on 2026-08-22.
-
-## Build runbook
-
-```mermaid
-graph TD
-    A[apt build deps] --> B[Clone ROCmFPX at pinned tag]
-    B --> C[Clone CK at pinned commit]
-    C --> D[Apply and verify CK patch]
-    D --> E[Generate ck/config.h: release gap]
-    E --> F[Run release build script]
-    F --> G[Build verification commands]
-    G --> H[Download and verify model]
-    H --> I[Smoke test]
-```
-
-### 1. Install build dependencies
-
-```
-sudo apt-get update
-sudo apt-get install -y build-essential cmake git ninja-build pkg-config libssl-dev
-```
-
-### 2. Clone the immutable release
-
-The local checkout is a git worktree of the pinned tag at
-`/home/dank/ciru-qwen38/ROCmFPX-kairic`.
-
-```
-git clone https://github.com/ciru-ai/ROCmFPX.git
-cd ROCmFPX
-git checkout kairic-edge-qwen38-27b-v1
-git status --short   # must print nothing
-```
-
-### 3. Prepare Composable Kernel
-
-```
-git clone https://github.com/ROCm/composable_kernel.git third_party/composable_kernel
-git -C third_party/composable_kernel checkout fdf4bb7fcc984811cef48ce817d89aac064b984a
-git -C third_party/composable_kernel apply ../../patches/composable-kernel-gfx1151-iu4.patch
-sha256sum patches/composable-kernel-gfx1151-iu4.patch   # ea57bca3793d...af5
-git -C third_party/composable_kernel diff --check
-```
-
-### 4. Generate `ck/config.h` (release gap)
-
-The pinned CK commit ships only `include/ck/config.h.in` (CMake template).
-The release script and the guide never generate `include/ck/config.h`. A
-clean build fails at the first CK include without it. Generate it with CK's
-own CMake configure, then copy it into the CK source include tree:
-
-```
-cmake -S third_party/composable_kernel -B third_party/composable_kernel/build-ck \
-  -G Ninja -DCMAKE_BUILD_TYPE=Release \
-  -DCMAKE_C_COMPILER=/usr/bin/gcc -DCMAKE_CXX_COMPILER=/usr/bin/g++ \
-  -DCMAKE_HIP_COMPILER="$ROCM_PATH/bin/amdclang++" \
-  -DCMAKE_PREFIX_PATH="$ROCM_PATH"
-cp third_party/composable_kernel/build-ck/include/ck/config.h \
-   third_party/composable_kernel/include/ck/config.h
-```
-
-Notes:
-
-- The CK configure can fail after `config.h` generation. The generated file
-  is still valid. Check `build-ck/include/ck/config.h` exists.
-- The `CK_USE_WMMA` flag inside `config.h` gates only CK instance-selection
-  code. The Kairic patched CK headers do not reference it. Local rebuilds
-  with the flag ON and OFF produce byte-identical binaries.
-- The file is untracked in CK git. A `git clean` in the CK tree removes it.
-  Regenerate it before any rebuild.
-- A full CK configure without a GPU target leaves `CK_USE_WMMA` undefined.
-  Pass `-DGPU_TARGETS=gfx1151` for the certified-style variant. Both
-  variants build an identical Kairic binary.
-
-### 5. Build
-
-The release script owns the certified CMake options.
-
-```
-export ROCM_PATH=<path to the 7.15 SDK _rocm_sdk_devel directory>
-JOBS="$(nproc)" scripts/build-kairic-edge-gfx1151.sh
-```
-
-Local `ROCM_PATH` is the split SDK layout inside the retired ling3-ciru
-venv:
-`/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_devel`.
-The serve script assembles `LD_LIBRARY_PATH` from the three split
-directories (`_rocm_sdk_devel`, `_rocm_sdk_libraries`, `_rocm_sdk_core`) so
-the 7.15 libraries precede the system ROCm 7.2.4.
-
-### 6. Build verification
-
-```
-./build-kairic/bin/llama-server --version
-./build-kairic/bin/llama-server --help | grep -A1 -- '--kairic-edge'
-ldd ./build-kairic/bin/llama-server | grep -E 'not found'   # must print nothing
-```
-
-Expected local output: `version: 79 (e97b324)`,
-`built with GNU 15.2.0 for Linux x86_64`.
-
-### 7. Download and verify model artifacts
-
-Download the four files from
-[jcbtc/Qwen3.8-27B-IU4-Kairic-Edge](https://huggingface.co/jcbtc/Qwen3.8-27B-IU4-Kairic-Edge)
-and verify the SHA-256 values in the table above.
-
-### 8. Smoke test
-
-Run the guide's smoke test: health, `/v1/models`, two identical
-`cache_prompt` completion requests, warm request must report restored
-prompt tokens. Inspect the server log for route fallback, swap activity,
-and HIP errors. Confirm the IU4 lane at boot:
-
-```
-{"record":"promptforge_init","mode":"iu4_ffn","wmma":"v_wmma_i32_16x16x16_iu4",...}
-{"record":"promptforge_gdn_init","projection":"qkvz_iu4_hadamard",...}
-```
-
-## Build reproducibility
-
-On 2026-08-22 a fresh rebuild followed the documented procedure from a
-clean CK tree (pinned commit, patch re-applied, `config.h` regenerated via
-CK configure). The rebuild produced a byte-identical binary:
-
-| Build | SHA-256 |
-|-------|---------|
-| Original (first deployment) | `d7625a0c14ac4158fd793c50ac0cefd8cdcbf76b5aca5cddafa09a353dc8f048` |
-| Fresh rebuild (clean CK tree) | `d7625a0c14ac4158fd793c50ac0cefd8cdcbf76b5aca5cddafa09a353dc8f048` |
-
-The two builds used different `config.h` variants (differ only in
-`/* #undef */` comment lines and the `CK_USE_WMMA` state). The identical
-output proves that variant difference has no effect on this build. The
-original manual `config.h` step is a neutral, minimal bridge over the
-release gap.
-
-The original build tree stays at
-`/home/dank/ciru-qwen38/ROCmFPX-kairic/build-kairic-orig/` until the next
-service restart confirms the fresh tree, then it is a deletion candidate
-(move to `.AGENTTRASH`).
-
-## Serving profile
-
-The service runs the pinned runner profile plus the local chat
-deviations below. Tunables live in
-`scripts/frank-dragoon.env`; flags in `scripts/serve-frank-dragoon.sh`.
-See [[deployment]] for the update process.
-
-| Setting | Value | Origin |
-|---------|-------|--------|
-| Bind | `0.0.0.0:52625` | Local (runner default `127.0.0.1:8080`) |
-| Context | 262144, 1 slot | Pinned runner |
-| Batch / ubatch | 2048 / 512 | Pinned runner (fastest measured prefill) |
-| Model loading | `--no-mmap` | Local (benchmarks: no perf difference) |
-| Threads | 16 target / 32 batch | Pinned runner |
-| KV | f16 target and draft | Pinned runner |
-| Prompt cache | `--cache-ram 8192 --cache-prompt --cache-idle-slots -ctxcp 32` | Pinned runner |
-| Kairic Edge | `--kairic-edge` | Pinned runner |
-| Speculative | `draft-mtp` depth 4, `p_split 0.10`, backend sampling | Pinned runner |
-| Server sampling | temp 0 / top-p 1 / top-k 0 / min-p 0 | Pinned runner |
-| Reasoning | on, deepseek format, budget -1, preserve_thinking | Local |
-| `LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH` | 0 | Local (runner: 1) |
-
-Local deviations and their reasons:
-
-- **Bind/port**: fleet convention; the runner binds localhost only.
-- **Reasoning on + deepseek**: thinking goes to `message.reasoning_content`,
-  matching the retired v3 service. `preserve_thinking=true` keeps thinking
-  tokens in the history across turns.
-- **`--no-mmap`**: benchmarks show no performance difference versus mmap.
-  Kept per operator preference (2026-08-22).
-- **Target greedy argmax fast path off**: the pinned runner sets
-  `LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=1`. That fast path fails closed on
-  any non-greedy request (400 error). The model card's own chat
-  instructions send temp 0.7 / top-p 0.8 / top-k 20 / presence_penalty
-  1.5, which the fast path rejects. Set to 0 for production chat on
-  2026-08-22. This does not affect the Kairic IU4 lane or the MTP draft
-  fast path (`LLAMA_MTP_CPU_ARGMAX_FASTPATH=1` stays on).
-- **`--kairic-edge` auto-injects ngram-mod** (n_match 24, n_min 64,
-  n_max 64). This is release design, not a misconfiguration.
-
-## Performance on this machine
-
-Benchmarks from 2026-08-22 on scratch port 7893, 512 forced tokens,
-greedy, production config unless noted. Stop the service before
-benchmarking (see AGENTS.md).
-
-| Context | Kairic prefill | Kairic decode | v3 prefill | v3 decode (200 tok) |
-|---------|----------------|---------------|------------|---------------------|
-| ~8k | 506 (runner) / 348 (prod) | 21.0 (runner) / 19.8 (prod) | 217.7 | 29.42 |
-| ~16k | 439 (runner) / 370 (prod) | 23.5 (runner) / 19.4 (prod) | 211.2 | 26.42 |
-| ~64k | 232.5 (prod) | 14.6 (prod) | 155.7 | 19.72 |
-
-Findings:
-
-- Prefill runs 1.5-2.3x faster than v3 at every depth.
-- Prose decode runs 15-26% slower than v3 at every depth.
-- `--no-mmap` makes no difference (19.89 vs 19.80 t/s decode). The
-  service now runs `--no-mmap` per operator preference.
-- `-c 262144` versus `-c 8192` makes no difference to decode.
-- Live service at ~116k context: 13.5-14.9 t/s, consistent with the 64k
-  bench. Long conversations run in the deep-context regime.
-- MTP draft acceptance on prose: rate 0.31-0.35, mean accepted length
-  2.6-3.0. Draft fast path active, zero failures.
-- The card's 47.73 t/s aggregate comes from a 164-task coding suite with
-  deterministic sampling and repeated coding context. It does not
-  represent prose chat. The card's natural-prose slice measured 34.88
-  t/s on the certified system.
-
-### Batch profile comparison (2026-08-22)
-
-Three batch profiles ran on the identical chat profile (fast path off,
-reasoning on, `--no-mmap`; the ub2048 row was the earlier mmap run, and
-mmap makes no difference):
-
-| Context | b2048/ub512 decode (cold/warm) | b4096/ub2048 decode | b4096/ub4096 decode | ub512 prefill cold | ub4096 prefill cold |
-|---------|--------------------------------|---------------------|---------------------|--------------------|---------------------|
-| ~8k | 19.6 / 18.7 | 19.8 / 19.4 | 22.7 / 19.8 | 524 | 266 |
-| ~16k | 21.8 / 21.5 | 19.8 / 18.8 | 18.1 / 22.5 | 485 | 258 |
-| ~64k | 13.7 / 15.7 | 14.6 / - | 14.1 / 13.8 | 323 | 203 |
-
-Ubatch is a prefill knob. Ubatch 512 gives the fastest prefill at every
-depth, 60-90% above ubatch 4096. Decode is identical across all three
-profiles within run-to-run noise. The IU4 lane accelerates 96-512-row
-prompt and verification shapes; ubatch 512 matches those shapes, large
-ubatches do not. Production runs the runner's qualified 2048/512.
-
-## Rollback to v3
-
-The v3 payload and binary stay on the machine for rollback.
-
-1. Stop the service: `sudo systemctl stop frank-dragoon`
-2. Restore the v3 scripts:
-   `git revert 4c43839 7aee1df 40d50fe` (or checkout the script pair from
-   commit `132e147`)
-3. Deploy: `FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw`
-4. Verify health and the `Output-K8` boot records.
-
-v3 locations: model `/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/`,
-binary `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`.
-Full v3 runbook: [[ciru-activefpx-promptforge]].
-
-## Citations
-
-- Model card: <https://huggingface.co/jcbtc/Qwen3.8-27B-IU4-Kairic-Edge>
-- Build and verification guide: <https://github.com/ciru-ai/ROCmFPX/blob/kairic-edge-qwen38-27b-v1/docs/kairic-edge-gfx1151.md>
-- Pinned release: <https://github.com/ciru-ai/ROCmFPX/tree/kairic-edge-qwen38-27b-v1>
-- Pinned runner: `scripts/run-kairic-edge-gfx1151.sh` in the release repo
-
-## See Also
-
-- [[ciru-activefpx-promptforge]] — retired v3 runtime, rollback target
-- [[deployment]] — deploy and update process
-- [[Dankclaw System Specs]] — hardware and memory model
-- [[qwen3.8-strix-optimizations]] — Strix Halo optimization history

serverAppConfigs-docs/LLM/kingjones-q8agent-rocmfpx.md

@@ -238,4 +238,4 @@ reads `--host/--port/--temp` as CLI flags, never as env vars.
   262144 env context) are in place. Proven serving context today is 131072 at
   `gl=45`. See the `set_rows` build bug section above.
 - Preceding model: Kairic Edge Qwen3.8-27B IU4 (deployed 2026-08-25 ~05:42,
-  retired the same day for the AGENT model). See [[kairic-edge]].
+  retired the same day for the AGENT model). See [[frank-dragoon]].

serverAppConfigs-docs/LLM/kingjones-strix-mtp.md

@@ -1,101 +0,0 @@
----
-type: reference
-title: kingjones Qwen3.8-27B ROCmFP4 STRIX MTP on Dankclaw
-description: kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Dense model with a standalone MTP Q4_0 draft head. frank-dragoon.
-tags:
-  - llama.cpp
-  - rocm
-  - dankclaw
-  - gfx1151
-  - kingjones
-  - rocmfp4
-  - mtp
-timestamp: 2026-08-24
----
-
-# kingjones Qwen3.8-27B ROCmFP4 STRIX MTP on Dankclaw
-
-> **RETIRED 2026-08-29.** This payload was `frank-dragoon` from 2026-08-25
-> until the Q8_0 AGENT experiment on 2026-08-29 (it was restored again
-> 2026-09-01 and served until 2026-08-30, when [[agentionai-fast-imatrix]]
-> took over). The Q8_0 model was retired because it crashed before
-> serving. Kept as the historical record of the Q6 AGENT model, standalone
-> MTP draft, Q8 projector, and PromptForge runtime.
-
-> This page documents a retired model and its runbook.
-
-`frank-dragoon` (port 7770) serves
-`Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP` from `kingjones777` on Hugging
-Face. It is a **dense** Qwen3.8-27B, abliterated (uncensored), quantized to
-the ROCmFP4 Q6 AGENT format (ftype 114). The MTP speculative head ships as
-a **separate** Q4_0 draft GGUF, so a bundled-head auto-detect does not apply
-here; the head is loaded with `--model-draft`.
-
-## Model Files
-
-The card ships six text files plus the vision projector. The service uses
-the Q6 AGENT file (requested for the fleet), the separate MTP draft head
-and the Q8 vision projector (`mmproj-Qwen3.8-27B-Q8_0.gguf`).
-
-| File | Size |
-|------|------|
-| `Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf` | 23.15 GiB |
-| `mtp-Qwen3.8-27B-Q4_0.gguf` (draft head) | 1.56 GiB |
-| `mmproj-Qwen3.8-27B-Q8_0.gguf` (vision) | 0.59 GiB |
-
-Store the files under `/home/dank/models/kingjones-Qwen3.8-27B-STRIX-MTP/`.
-
-## Speculative Decoding
-
-Dense model, so the finding measured on the dense Qwen3.8 family holds:
-**MTP alone at `--spec-draft-n-max 4` is the knee**. ngram under MTP hurts
-(measured -4 to -5 t/s on the v3; see [[ciru-activefpx-promptforge]]). The
-draft head is a separate file loaded via:
-
-```
---spec-type draft-mtp
---model-draft mtp-Qwen3.8-27B-Q4_0.gguf
---spec-draft-n-max 4
---spec-draft-n-min 0
---spec-draft-p-min 0.0
-```
-
-The draft head loads on `ROCm1` with full offload (`--spec-draft-ngl 999`)
-and f16 KV.
-
-## Launch Excerpt
-
-The key flags on the CIRU ROCmFPX runtime:
-
-```
-llama-server \
-  -m Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf \
-  --mmproj mmproj-Qwen3.8-27B-Q8_0.gguf \
-  --chat-template-file chat-template.jinja \
-  --chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true}' \
-  -ngl 999 -dev ROCm0 -c 196608 -np 1 \
-  -b 8192 -ub 8192 -ctxcp 32 \
-  --spec-type draft-mtp --model-draft mtp-Qwen3.8-27B-Q4_0.gguf \
-  --spec-draft-ngl 999 --spec-draft-n-max 4 \
-  -fa on --no-mmap --cache-type-k f16 --cache-type-v f16
-```
-
-## Known Issues
-
-1. **GPU only**: Vulkan, CUDA, and CPU cannot load these files.
-2. `--spec-draft-n-max` defaults to 16; set 4.
-3. `reasoning_effort: "none"` throws in the Qwen API. Use
-   `enable_thinking: false` instead.
-4. Small `max_tokens` with thinking on returns empty `content`. The Sharp
-   template drives thinking on by default (effort medium).
-5. `preserve_thinking:true` (in `chat_template_kwargs`) keeps `<think>`
-   blocks in recorded assistant messages. `preserve_thinking:false` strips
-   them. See [[deployment]] for the chat_format/reasoning_format pairing
-   (`peg-native` + `deepseek`) the Pi agent expects from this endpoint.
-
-## See Also
-
-- [[deployment]] — deploy and update process
-- [[ciru-activefpx-promptforge]] — the retired v3 page
-- [[Dankclaw System Specs]] — the box
-- [[frank-redo-2026-08-17]] — fleet topology history

serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md

@@ -1,322 +0,0 @@
----
-type: reference
-title: kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw (prior, rollback target)
-description: PRIOR frank-dragoon runbook and rollback target (2026-08-30 → 2026-09-11). kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF served by the card's pinned kingjones30/ROCmFPX fork. 177B params (56B MoE + 51.2B PLE streamed via mmap), single 113.47 GiB GGUF, 401408 ctx x 2 slots.
-tags:
-  - llama.cpp
-  - rocm
-  - dankclaw
-  - gfx1151
-  - kingjones
-  - rocmfp4
-  - strix-halo
-  - qwen4exp
-timestamp: 2026-08-30
----
-
-# kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw
-
-> **PRIOR / ROLLBACK TARGET since 2026-09-11** — frank-dragoon moved to
-> [[halogen-flash-server]] (peonist-ai halogen-flash-server 0.5.8 serving
-> Qwen3.8-Flash-Next w4b), which upstream measures at roughly 4x this
-> runtime's prefill. Everything below documents the llama.cpp era and stays
-> accurate for a rollback: the payload is still at
-> `/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX`, the binary
-> at `/home/dank/GitPackages/kingjones-ROCmFPX/build`, and the unit, env and
-> serve script are staged in `serverAppConfigs/.AGENTTRASH/` (`.llama.cpp`).
-> Rolling back also requires removing the halogen Quadlet from
-> `~/.config/containers/systemd/` so it stops claiming port 7770.
->
-> Ran 2026-08-30 → 2026-09-11 on the model card's **pinned runtime**: the
-> `kingjones30/ROCmFPX` fork. Text-only tier; the agentionai f16 vision
-> tower was paired for mmproj and loads clean. Speculative decoding is OFF
-> (the card ships no spec flags; the Flash-Next MTP heads do not load on
-> this loader family — see MTP Draft Head).
-
-`frank-dragoon` (port 7770) serves
-`Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf` (single file, 113.47 GiB)
-from `/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/`
-via `scripts/serve-frank-dragoon.sh` + `scripts/frank-dragoon.env`
-and the Qwen-Sharp chat template.
-
-## Model Architecture
-
-Base model: Qwen3.8-Flash-Next (177B parameters total).
-
-| Component | Params | Details |
-|-----------|--------|---------|
-| MoE experts | 125.8B | 512 experts, 10 used per token |
-| Attention | dense | 48 layers (12 attn + 36 SSM) |
-| PLE table | 51.2B | `per_layer_token_embd`, Q8_0 |
-| lm head | ~0.3% | Q6_K |
-
-The PLE (per-layer embedding) n-gram table is the only thing that differs
-between STRIX, STRIX_LEAN, and FAST tiers. Everything else is byte-identical.
-
-## File Details
-
-| Field | Value |
-|-------|-------|
-| File | `Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf` |
-| Size | 113.47 GiB |
-| Format | Single GGUF (not sharded) |
-| file_type | 105 (`Q4_0_ROCMFP4_STRIX`) |
-| General | 113.47 GiB, 5.51 bpw |
-
-Tensor composition:
-
-| Tensor group | Type |
-|---|---|
-| MoE expert weights (`ffn_*_exps`, 144) | TYPE_101 (ROCmFP4, 4.251 bpw) |
-| Shared expert (`ffn_*_shexp`, 144) | TYPE_101 |
-| Attention (`attn_*`, 120) | half TYPE_100, half TYPE_101 |
-| PLE (`per_layer_token_embd.weight`) | Q8_0 — 50.66 GiB |
-| `token_embd.weight` | Q6_K |
-| `output.weight` (lm head) | Q6_K |
-| Norms / biases | F32 |
-
-## File Type Reference
-
-| Tier | file_type | PLE table | Total size |
-|------|-----------|-----------|------------|
-| FAST | TYPE_106 | 25.34 GiB | 87.94 GiB |
-| STRIX_LEAN | TYPE_105 (old STRIX_LEAN) | 35.76 GiB | 98.49 GiB |
-| **STRIX** | **105 (`Q4_0_ROCMFP4_STRIX`)** | **50.66 GiB** | **113.47 GiB** |
-
-TYPE_100 = 4.506 bpw, TYPE_101 = 4.251 bpw (measured exactly by solving
-three files for two unknowns).
-
-## Memory
-
-The PLE table never enters GPU VRAM. It streams from the file through the OS
-page cache.
-
-| Context | GTT at load | GTT after prompt | GPU resident |
-|---------|-------------|-------------------|-------------|
-| 8,192 | 63.3 GiB | 63.6 GiB | ~63 GiB |
-| 32,768 | 64.1 GiB | 64.8 GiB | |
-| 65,536 | 65.1 GiB | 66.2 GiB | |
-| 131,072 | 67.2 GiB | 69.1 GiB | ~69 GiB |
-
-~63 GiB is the non-PLE mass plus compute buffers. **The PLE table does not
-grow GPU memory regardless of context depth.**
-
-Three critical consequences:
-
-1. **`--no-mmap` gets you OOM-killed.** It forces the PLE into anonymous memory
-   (neither file-backed nor reclaimable). The cgroup killer takes the process
-   with nothing in the server log — only `dmesg` shows the evidence.
-2. **Don't force PLE to CPU with `-ot`.** Measured generation dropped from
-   ~23 to 13.4 tok/s. The kernel streams it better than any offload hack.
-3. **A bigger tier costs disk, not GPU memory.** If you have storage, take
-   STRIX.
-
-## Performance
-
-Measured on Ryzen AI MAX+ 395 (gfx1151, ROCm 7.2.4), 49/49 offload. Prompts
-sized to 85% of context window:
-
-| Context | Prompt tokens | pp tok/s | gen tok/s |
-|---------|--------------|----------|-----------|
-| 8,192 | 6,963 | 279 | **22.02** |
-| 32,768 | 27,852 | 264 | **20.52** |
-| 65,536 | 55,705 | 234 | **18.23** |
-| 131,072 | 111,411 | 185 | **15.34** |
-
-Short prompts (~3,300 tokens): 381 tok/s prompt processing, 22.7 tok/s
-generation (median of 3 runs).
-## Benchmark: threads-batch tuning
-
-Sweeped `--threads-batch` (16, 32, 48, 64, 80, 96, 128) with a 64K-token
-prompt via HTTP API (`max_tokens=1`). Fresh server start per value.
-
-| threads-batch | prefill tok/s | wall time |
-|---------------|--------------|-----------|
-| **16** | **373.5** | 62.9s |
-| 32 | 354.4 | 66.3s |
-| 48 | 344.1 | 68.3s |
-| 64 | 344.7 | 68.2s |
-| 80 | 344.4 | 68.2s |
-| 96 | 357.1 | 65.8s |
-| 128 | 321.1 | 73.2s |
-
-**Optimal: `--threads-batch 16`** (373.5 tok/s). Lower thread batching wins
-for prefill on this model -- fewer threads compete for GPU resources during
-the prompt processing phase. The default of 32 is 5.1% slower. Above 64,
-performance plateaus then declines (128 is 14% worse than optimal).
-
-## Benchmark: decode throughput
-
-Sweeped `--threads-batch` (8, 16, 32) with a ~23.5K-token code prompt and
-256 generated tokens via HTTP API. Fresh server start per value.
-
-| threads-batch | gen tokens | wall time | gen tok/s |
-|---------------|-----------|-----------|-----------|
-| 16 | 256 | 74.8s | 3.4 |
-| 32 | 256 | 75.0s | 3.4 |
-| 8 | 256 | 76.6s | 3.3 |
-
-**Decode throughput is essentially flat across all tested values.** TB=16 and
-TB=32 are identical at 3.4 tok/s; TB=8 is marginally slower at 3.3 tok/s.
-Combined with the 5.1% prefill advantage at lower thread counts, `--threads-batch
-16` is the optimal setting for both phases.
-
-## Runtime build (kingjones30/ROCmFPX)
-
-The card pins its own fork: [kingjones30/ROCmFPX](https://github.com/kingjones30/ROCmFPX)
-— charlie12345/ROCmFPX plus the `qwen4exp-on-rocmfpx-d3ca537` patch, so
-ROCmFP4 tensor types and the qwen4exp arch live in one tree. Built at
-`/home/dank/GitPackages/kingjones-ROCmFPX/build` from commit `36e9acd`
-(2026-08-27 card revision) with the card's recipe:
-
-```bash
-git clone https://github.com/kingjones30/ROCmFPX.git
-cd ROCmFPX
-cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release -DLLAMA_BUILD_WEBUI=OFF
-cmake --build build --target llama-server llama-quantize -j16
-```
-
-- **`-DLLAMA_BUILD_WEBUI=OFF` is required on this box**: the tree
-  (post-WebUI llama.cpp) provisions embedded UI assets via npm or the
-  HF llama-ui bucket; npm is absent here and the bucket download
-  half-failed, aborting the link at 98%. The API server is unaffected.
-  All other compile work succeeded with zero errors, confirming the
-  card's clean-clone claim.
-- All 35 flags our serve script passes were verified present in this
-  tree (`--spec-type`, `--cache-idle-slots`, `--ctx-checkpoints`,
-  `--reasoning-budget`, `--mmproj`, …).
-
-## Serving Profile
-
-Deployed args (`scripts/frank-dragoon.env` — our validated
-profile, unchanged from the FAST deployment except the binary and model):
-
-```
-llama-server \
-  --model Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf \
-  --host 0.0.0.0 --port 7770 --jinja \
-  --device ROCm0 --gpu-layers 999 \
-  --fit off --flash-attn on \
-  --ctx-size 401408 --parallel 2 \
-  --batch-size 4096 --ubatch-size 512 \
-  --cache-type-k f16 --cache-type-v f16 \
-  --threads 16 --threads-batch 16 \
-  --ctx-checkpoints 16 --cache-ram 8192 \
-  --cache-prompt --cache-idle-slots --metrics \
-  --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 \
-  --presence-penalty 1.5 \
-  --reasoning on --reasoning-format deepseek --reasoning-budget -1 \
-  --chat-template-file chat-template.jinja \
-  --chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}' \
-  --mmproj mmproj-Qwen3.8-Flash-Next-f16.gguf
-```
-
-Key choices:
-
-- **`--fit off`** — auto-fit segfaults on this model's qwen4exp
-  hyper-connection/indexer configuration.
-- **mmap (default)** — do NOT use `--no-mmap`. See memory section.
-- **`--flash-attn on` works unpatched on this model** — verified live
-  2026-08-30 (graph built, model served). The QSA F16 `ggml_fill`
-  assert that forced [[agentionai-fast-imatrix]] onto the patched
-  charlie tree did not fire for the kingjones STRIX file; if a future
-  build aborts with `GGML_ASSERT(a-type == GGML_TYPE_F32)`, port
-  `scripts/patches/qwen4exp-qsa-f32-mask.patch` into this tree.
-- **`--ctx-size 401408 --parallel 2`** (200704/slot) — carried from the
-  FAST deployment; the card's own ladder proves the window is nearly
-  free (QSA caps KV: ~8 GiB total 8k→256k). Verified live at depth:
-  a 111k-token prompt prefills at ~230 t/s while a second slot decodes.
-- **`--ubatch-size 512`** — card re-measurement: -ub 1024/2048/4096
-  show no effect; 512 is already right on this hardware.
-- **`--mmproj`** — the STRIX repo ships no vision tower; the agentionai
-  FAST package's f16 mmproj (same base model) is paired and loads
-  (one benign `warmup: CLIP graph uses unsupported operators` warning).
-- **No spec flags** — speculative decoding is off; see below.
-
-## MTP Draft Head — not loadable on this loader family
-
-The previous draft of this runbook ran
-[EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF](https://huggingface.co/EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF)
-(Q8_0, 4.1 GB) as `--model-draft`. It does **not** load: the qwen4exp
-loader requires a full per-layer `blk.<il>.hc_attn_norm.weight` set and
-every Flash-Next-era head carries only `blk.48` (verified 3x against
-multiple mains on 2026-08-28/29, see [[agentionai-fast-imatrix]] known
-issues). The service therefore runs with no speculative decoding — the
-card's own Usage block ships none. The card's published +50% decode
-figure was measured with `Aristo94/EngramHalo.cpp`, a fork we do not
-build. Revisit only if a head with the complete hyper-connection norm
-set (or a loader that tolerates partial draft tensors) appears.
-
-
-## Validation (2026-08-30 swap)
-
-1. Integrity: on-host file `sha256sum` = `c6770d7442a06bf1d78edf28cec83e1ec93afdd34664c23ff898807b6b9349fa`,
-   byte-identical to the HF LFS oid. No re-download needed.
-2. Load: systemd unit healthy; model loaded in ~225 s; health 200.
-3. Generation: `/v1/chat/completions` returns correct
-   `reasoning_content` + content through the Sharp template (deepseek
-   format, `preserve_reasoning`).
-4. Real load: an operator session's ~111k-token prompt prefilled at
-   213–232 t/s under the new runtime, matching the card's 131k rung
-   (196 t/s), with checkpoints + QSA KV streaming live.
-
-## Deployment
-
-### Download
-
-```bash
-# on Dankclaw
-TOKEN=$(cat ~/.cache/huggingface/token)
-DIR=/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX
-mkdir -p "$DIR"
-curl -fsSL -C - -o "$DIR/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf" \
-  --retry 5 --retry-delay 3 --retry-max-time 7200 \
-  -H "Authorization: Bearer $TOKEN" \
-  "https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF/resolve/main/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf"
-```
-
-The file is 113.47 GiB. Allow ~20-30 minutes for download depending on network.
-
-### Swap procedure (as run 2026-08-30)
-
-1. Build the runtime (see Runtime build) BEFORE touching the service.
-2. Point the model file's directory at the canonical path (a plain `mv`
-   — never delete a 113 GiB payload to re-fetch it).
-3. Edit `scripts/frank-dragoon.env` (+ serve script defaults) via
-   git: commit, push, `./scripts/deploy.sh dankclaw --service
-   frank-dragoon`.
-4. Verify health: `curl http://192.168.0.143:7770/health` and a real
-   `/v1/chat/completions` probe.
-
-Rollback: the FAST deployment is one env commit away (`LLAMA_BIN`,
-`MODEL_DIR`, `MODEL_FILE`, `MMPROJ_PATH`); the charlie `build-qwen38`
-binary and the agentionai payload remain on disk.
-
-### Config files
-
-- Env: `scripts/frank-dragoon.env`
-- Serve: `scripts/serve-frank-dragoon.sh`
-- Chat template: `scripts/chat-template.jinja`
-
-## Known Issues
-
-1. **Pinned fork required.** Stock llama.cpp and plain charlie12345/
-   ROCmFPX each lack half the tree (qwen4exp arch / ROCmFP4 types). Use
-   kingjones30/ROCmFPX, or the equivalent merged charlie tree
-   (`/home/dank/GitPackages/ROCmFPX`, PR #27742 commits).
-2. **Single-file payload.** Not sharded. Use `hf download` or a resumable
-   curl. Browser download unreliable for 113 GiB.
-3. **Model load is slow.** A few minutes cold for the 113 GiB file with a
-   fresh KV cache. Health returns 503 while loading. Do not kill the process.
-   `deploy.sh` waits up to 20 minutes.
-4. **Kernel spam about `amdgpu: SVM mapping failed`** is a known documented
-   symptom, not root cause. Investigate other causes before assuming memory
-   pressure.
-
-## Citations
-
-- [kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF)
-- [Qwen/Qwen3.8-Flash-Next base model](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)
-- [kingjones30/ROCmFPX](https://github.com/kingjones30/ROCmFPX) — the card's pinned fork (charlie12345/ROCmFPX + [qwen4exp-on-rocmfpx-d3ca537.patch](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF/resolve/main/qwen4exp-on-rocmfpx-d3ca537.patch), mirrored in the HF repo)
-- [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) — upstream fork defining the ROCmFP4 formats; our rollback tree with PR #27742 merged
-- [llama.cpp PR #27739](https://github.com/ggml-org/llama.cpp/pull/27739) — MTP draft head support

serverAppConfigs-docs/LLM/ling-3.0-flash-trial.md

@@ -8,7 +8,7 @@ timestamp: 2026-09-03
 
 # Ling-3.0-flash Q6_K Trial on Dankclaw
 
-> **RETIRED 2026-08-17.** The `frank-dragoon-ling` unit is disabled under the boot policy. At this page's retirement date `frank-dragoon` ran the Qwen3.8-27B PP head (since superseded by the STRIX deployment — see [[kingjones-strix-rocmfp4]]), not the Qwen3.6-27B config described in the (obsolete) switch-back section below.
+> **RETIRED 2026-08-17.** The `frank-dragoon-ling` unit is disabled under the boot policy. At this page's retirement date `frank-dragoon` ran the Qwen3.8-27B PP head (since superseded by the STRIX deployment — see [[frank-dragoon]]), not the Qwen3.6-27B config described in the (obsolete) switch-back section below.
 
 ## Status
 
@@ -111,6 +111,6 @@ re-enable `frank-dragoon-ling` from the repo unit
 
 ## See Also
 
-- [[frank-dragoon-benchmark]]
+- [[frank-dragoon]]
 - [[DFlash DSpark Speculative Decoding]]
 - [[deployment]]

serverAppConfigs-docs/LLM/qwen3.8-spec-matrix.md

@@ -162,6 +162,6 @@ q8_0/f16 KV.
 
 ## See Also
 
-- [[frank-dragoon-benchmark]]
+- [[frank-dragoon]]
 - [[ling-3.0-flash-trial]]
 - [[deployment]]

serverAppConfigs-docs/LLM/qwen3.8-strix-optimizations.md

@@ -15,7 +15,8 @@ and the test matrix planned for 2026-08-16 morning.
 
 Related pages:
 - [Speculative-decoding matrix](qwen3.8-spec-matrix.md)
-- [Backend and spec benchmark](frank-dragoon-benchmark.md)
+- [[frank-dragoon]] — the endpoint this host became; its Eras and Laws carry the
+  backend and spec-decode measurements.
 - [Dankclaw Improvements](Dankclaw%20Improvements.md) (OS-level items)
 - [Pipeline Parallelism: Qwen3.8 PP Deployment](Pipeline%20Parallelism/Qwen3.8%20PP%20Deployment.md)
 
@@ -253,5 +254,5 @@ production @16k) with the same 8-bit model.
   `-dio` for RPC).
 - llama.cpp b10433 source: `src/llama.cpp` (device selection),
   `common/speculative.cpp` (MTP/ngram-mod), `ggml/src/ggml-rpc` (protocol).
-- Local benchmarks: [frank-dragoon-benchmark](frank-dragoon-benchmark.md),
+- Local benchmarks: the measurement record lives on [[frank-dragoon]] (Eras and
-  [qwen3.8-spec-matrix](qwen3.8-spec-matrix.md).
+  Laws) and in [qwen3.8-spec-matrix](qwen3.8-spec-matrix.md).

serverAppConfigs-docs/LLM/ubatch-benchmark-large.md

@@ -53,4 +53,4 @@ from the new harness here after a clean run.
 
 - [LLM fleet index](index.md)
 - [Deployment](deployment.md)
-- [Active kingjones runbook](kingjones-strix-mtp.md)
+- [[frank-dragoon]] — the kingjones MTP era this sweep ran against is an Eras row there (its page was folded in 2026-10-18)

serverAppConfigs-docs/LLM/vLLM Boot Forensics and Config Interrogation.md

@@ -203,5 +203,5 @@ clipping at depth) starts with sections 1–3 here before touching anything.
 ## Related
 
 - [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan arithmetic behind OOMs
-- [[dankserver-qwen38-fp8-expert-offload-tuning]] — deployed shape and history
+- [[frank-archon]] — deployed shape and history
-- [[frank-dragoon-benchmark]] and [[frank-zealot-benchmark]] — probe/ladder hygiene on live hosts
+- [[frank-dragoon]] and [[frank-zealot]] — probe/ladder hygiene on live hosts

serverAppConfigs-docs/LLM/vLLM Porting and Compile Triage.md

@@ -326,5 +326,5 @@ consumed at `v1/worker/gpu/cudagraph_utils.py:621`).
 
 - [[vLLM Boot Forensics and Config Interrogation]] — capture and contracts
 - [[RDNA4 Container Serving]] — the serving stack being ported onto
-- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the live deployment
+- [[frank-archon]] — the live deployment
-- [[halogen-flash-server]] — the retired w4b-era endpoint and its profiling notes
+- [[frank-dragoon]] — the retired w4b-era endpoint and its profiling notes

serverAppConfigs-docs/LLM/x4-pair-experiment.md

@@ -248,8 +248,8 @@ distinguishes a real split from a mislabelled sweep.
 ## Conclusion
 
 The fleet's answer is unchanged — **two independent endpoints**
-([[halogen-flash-server]] on the iGPU via halogen,
+([[frank-dragoon]] on the iGPU via halogen,
-[[frank-zealot-qwen38-gsq-rco]] holding a whole 27B in the 3090's 24 GB) — but two
+[[frank-zealot]] holding a whole 27B in the 3090's 24 GB) — but two
 claims from the first round of this experiment are now corrected:
 
 1. **Cross-vendor layer split is possible** (measured: CUDA0 + Vulkan1, 27.23

serverAppConfigs-docs/Research/Llama Cpp Code Diff.md

@@ -211,7 +211,7 @@ a packing transform on the K-family — and is likewise a modifier, not a quant.
 ## See Also
 - [[Llama Cpp Runtimes]] — positioning + quantization overview
 - [[dealign-ornith-crack]] — the CRACK MoE on Dankserver
-- [[kingjones-strix-mtp]] — the ROCmFP4 model on Dankclaw
+- [[frank-dragoon]] — the endpoint the Dankclaw ROCmFP4 eras became (their payloads and shapes are Eras rows there)
 - [[Quantization by Hardware Architecture]] — quant vs hardware
 
 ## Citations

serverAppConfigs-docs/Research/Llama Cpp Runtimes.md

@@ -125,7 +125,7 @@ Serves **kingjones Qwen3.8-27B ROCmFP4 STRIX MTP** (the production build).
 ## See Also
 - [[Llama Cpp Code Diff]] — the code-level quantization diff + comparison table
 - [[dealign-ornith-crack]] — the CRACK model on Dankserver
-- [[kingjones-strix-mtp]] — the ROCmFP4 model on Dankclaw
+- [[frank-dragoon]] — the endpoint the Dankclaw ROCmFP4 eras became (their payloads and shapes are Eras rows there)
 - [[Quantization by Hardware Architecture]] — which quant fits which GPU/CPU
 - [[SGLang]] — the structured-generation serving engine (the other lane)
 

serverAppConfigs-docs/Research/SGLang.md

@@ -142,7 +142,7 @@ SGLang-on-ROCm is supported (Docker/pip), but:
 ## See Also
 - [[Llama Cpp Runtimes]] / [[Llama Cpp Code Diff]] — the llama.cpp side of the trio
 - [[dealign-ornith-crack]] — the CRACK MoE on Dankserver
-- [[kingjones-strix-mtp]] — the ROCmFP4 MTP model on Dankclaw
+- [[frank-dragoon]] — the endpoint the Dankclaw ROCmFP4 MTP era became (an Eras row there)
 - [[Quantization by Hardware Architecture]]
 
 ## Citations

serverAppConfigs-docs/index.md

@@ -1,566 +1,235 @@
-# Server App Configs
+# serverAppConfigs — knowledge base
 
-Podman deployment configuration for two homelab servers: Dankserver and Dankclaw.
+One fact, one owner. A page owns exactly one thing and everywhere else links
+to it with a wikilink. Shape, payload, eras, laws and rollback for an inference
+endpoint live on its endpoint page; `scripts/<service>.env` is the authority for
+every number, and a page that disagrees with it is the bug. Payload swaps are
+rows in an endpoint page's **Eras** table, never pages of their own. This index
+is a locator: one row, one clause, no numbers.
 
-This repository is the single source of truth for both hosts. It tracks the
+## Inference endpoints
-container stacks and the LLM inference fleet. Edits flow through this repo,
-then deploy to the targets with `scripts/deploy.sh` (one script per target
-covers both workloads).
 
-> 2026-08-22: the former `vllmScripts` repository (LLM fleet) is merged
+| Page | Owns |
-> into this repo with its full git history. Its knowledge base now lives at
+|---|---|
-> [serverAppConfigs-docs/LLM/index.md](LLM/index.md).
+| [[Frank TTS]] | frank-tts is the fleet's cloned-voice speech endpoint on Dankclaw :7772 — the one voice surface consumers address, in… |
+| [[frank-archon]] | The one frank-archon page: the live radiance shape and harness contract (read off scripts/frank-archon.env), the Eras… |
+| [[frank-dragoon]] | The one runbook for frank-dragoon, the Strix Halo iGPU endpoint (192.168.0.143:7770): current shape from the env,… |
+| [[frank-zealot]] | The one frank-zealot reference: the live shape on Dankclaw's RTX 3090 (:7771) exactly as scripts/frank-zealot.env… |
 
-## Hosts
+## Retired endpoints
 
-| Host | IP | Role | OS |
+These services no longer exist, so no live page owns them; rollback provenance
-|------|----|------|----|
+is inside each page. Every payload swap of a LIVE endpoint is an Eras row on
-| Dankserver | 192.168.0.235 | Media & library stack | Ubuntu 24.04 |
+that endpoint's page, not a page.
-| Dankclaw | 192.168.0.143 | AI, search & development stack | Ubuntu 26.04 |
 
-## Stacks
+| Page | Owns |
+|---|---|
+| [[frank-medium-batch-ubatch]] | llama-bench sweep of --batch-size against --ubatch-size on frank-medium (2x Radeon AI PRO R9700, layer split) at the… |
+| [[frank-medium-llamacpp-dflash2]] | The former ACTIVE frank-medium deployment since 2026-09-16 (model swapped 2026-09-19) — mainline llama.cpp on… |
+| [[frank-medium-model-swap]] | How to swap the checkpoint behind the frank-medium endpoint (Dankserver :7770, vllm-radiance TP=2) — the streamlining… |
+| [[frank-medium-radiance-mxfp4]] | ACTIVE frank-medium runbook — vllm-radiance serving hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 (ParoQuant MXFP6 W4A8,… |
+| [[frank-medium-rocmfpx]] | charlie12345/ROCmFPX llama.cpp serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF as frank-medium on Dankserver… |
+| [[frank-npu]] | The retired FastFlowLM endpoint on Dankclaw's XDNA2 NPU — what it served, why it died on 2026-09-11 with the… |
+| [[frank-redo-2026-08-17]] | This page records the full redo of the inference fleet on 2026-08-17. |
+| [[ikllama-ornith-dankserver]] | ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-zealot on the RTX 3090. Q8_0 model, KV K default f16 / V… |
+| [[kingjones-q8agent-rocmfpx]] | **Status (historical, 2026-09-13):** was to deploy on Dankclaw. |
+| [[qwen3.8-spec-matrix]] | This page records the speculative-decoding benchmark history and the |
 
-- [[Dankserver Stack]] — RomM, Copyparty, Calibre, Calibre-Web, Caddy
+## Dankclaw services (192.168.0.143)
-- [[Dankclaw Stack]] — SearXNG (via Gluetun), ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Caddy
 
-## Architecture
+| Page | Owns |
+|---|---|
+| [[ComfyUI]] | ComfyUI is a node-based AI image generation engine. |
+| [[ComfyUI Qwen-Image Workflows]] | Qwen-Image 2.1 workflows on Dankclaw: two proven txt2img workflows (art + legible text), the API recipe, and which… |
+| [[Dankclaw Stack]] | Service inventory and routing overview for the Dankclaw AI, development, and observability host. |
+| [[Frank TTS Engine]] | frank-tts-engine is the arm A TTS engine on Dankclaw: qwentts.cpp's resident tts-server with the 1.7 B base talker,… |
+| [[Gitea]] | Gitea deployment on Dankclaw with its web UI reverse-proxied by the canonical Caddy service. |
+| [[Gluetun]] | Gluetun is a lightweight VPN client container. |
+| [[Hermes]] | Hermes is the self-improving AI agent from Nous Research. |
+| [[Hermes-Dashboard]] | The Hermes web dashboard is a browser UI for managing the agent. |
+| [[Hermes-WebUI]] | The hermes-webui container on Dankclaw — the API server that the Hermex iPhone app talks to, reachable over an SSH… |
+| [[Marreta]] | The content-cleaning proxy (manualdousuario/marreta, Laravel/serversideup fpm-nginx) deployed as a rootless Quadlet… |
+| [[Open Notebook]] | open-notebook is the fleet's podcast generator on Dankclaw: notes and sources in, a multi-speaker episode out —… |
+| [[OpenWebUI]] | OpenWebUI is a self-hosted chat interface for LLMs. |
+| [[Postgres]] | PostgreSQL is the database backend for Gitea. |
+| [[SearXNG]] | SearXNG is a privacy-respecting metasearch engine. |
+| [[SearXNG Providers]] | How Google and other upstream search engines get blocked behind the Gluetun |
+| [[Seerr]] | Seerr is the unified request manager for the *arr stack (the maintained `seerr-team/seerr` successor to the archived… |
+| [[Swap Configuration]] | The host swapfile on Dankclaw — 64 GiB at /swap.img on ext4, fstab-driven via the generated swap.img.swap unit.… |
+| [[Valkey]] | Valkey is the cache and limiter store for SearXNG. |
+| [[Voice Packs]] | How a frank-tts voice pack is built on Dankclaw: the three files every pack must carry, the 10-second source-material… |
 
-- [[Network Architecture]] — Caddy reverse proxy, hostname routing, Gluetun VPN path
+## Dankserver services (192.168.0.235)
-- [[Storage Policy]] — bind mounts, data locations, ownership
-- [[Deployment Flow]] — service-scoped git deploy, target map, and health checks
-- [[Testing Policy]] — the test seams and how to run them
-- [[E2E Suite]] — the Playwright suite that drives the live guard chain
-- [[Container Network Namespaces]] — which local services are addressable from each Podman network namespace on Dankclaw
 
-## Dankserver Storage
+| Page | Owns |
+|---|---|
+| [[Calibre]] | Calibre is the e-book library manager that backs Calibre-Web. |
+| [[Calibre-Web]] | Calibre-Web is a web application that browses, reads, and downloads eBooks from an existing Calibre library database. |
+| [[Copyparty]] | Copyparty is a lightweight file server and media streamer that runs as a single Python binary. |
+| [[Dankserver Stack]] | Dankserver (192.168.0.235) runs the media and library stack. |
+| [[Entropy Volume]] | The Dankserver entropy volume is a RAID-backed LUKS filesystem mounted at |
+| [[Jellyfin]] | Jellyfin media server on Dankserver — quadlet unit, entropy-volume data layout, ports, client access, and… |
+| [[mariadb]] | MariaDB 11 database backend for RomM on Dankserver — quadlet unit, network alias, env-file secret handling, and… |
+| [[Plex]] | Plex Media Server is a **containerized media server on Dankserver**: a podman |
+| [[redis]] | Redis 7-alpine cache and task-queue backend for RomM on Dankserver — quadlet unit, network alias topology, and backup… |
+| [[RomM]] | RomM (ROM Manager) is a self-hosted ROM library manager and player. |
+| [[RomM-Scraping]] | This guide covers metadata providers, cover art, ROM naming conventions, emulator integration, and the scraping… |
+| [[Tautulli]] | Tautulli is the statistics, monitoring, and notification companion for |
 
-- [[Entropy Volume]] — command-line unlock and mount helper
+## FrankBook-Pro (192.168.0.112)
 
-## Dankserver Services
+| Page | Owns |
+|---|---|
+| [[ARR-Stack]] | The *arr automation stack (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) — FrankBook-Pro only, every… |
+| [[lidarr]] | Lidarr music manager in the *arr stack on FrankBook-Pro — live quadlet deployment, ports, volumes, and wiring facts. |
+| [[prowlarr]] | Prowlarr indexer-management layer for the *arr stack on FrankBook-Pro — feed routes, sync model, API surface, and… |
+| [[qbittorrent]] | qBittorrent download client for the *arr stack — pinned 4.6.7 on FrankBook-Pro, shares the Gluetun WireGuard netns,… |
+| [[radarr]] | Radarr movie manager in the *arr stack on FrankBook-Pro — live quadlet deployment, ports, volumes, and wiring facts. |
+| [[readarr]] | Readarr ebook/audiobook manager in the *arr stack on FrankBook-Pro — retired upstream, pinned image, parked qB… |
+| [[sonarr]] | Sonarr TV manager in the *arr stack on FrankBook-Pro — live quadlet deployment, ports, volumes, and wiring facts. |
 
-- [[RomM]] — retro game manager
+## Workstations
-- [[RomM-Scraping]] — metadata providers, cover art, naming conventions
-- [[Copyparty]] — file sharing and streaming
-- [[Jellyfin]] — media server (movies, TV, music)
-- [[Calibre]] — e-book library engine
-- [[Calibre-Web]] — e-book web frontend
-- [[Plex]] — media server on Dankserver (lscr.io/linuxserver/plex container via
-  quadlet; CPU decode only — no GPU passthrough)
-- [[Tautulli]] — Plex statistics and notifications (v2.18.1-ls245, quadlet unit)
-- [[mariadb]] — RomM database backend
-- [[redis]] — RomM cache/queue backend
 
-## FrankBook-Pro Services
+| Page | Owns |
+|---|---|
+| [[Noctalia and Niri]] | The noctalia v5 desktop shell on niri (workstation + blankpad) — config layers, remote IPC over ssh, custom-palette… |
 
-The *arr stack and qBittorrent run on **FrankBook-Pro** (192.168.0.112, the
+## Cross-host infrastructure
-macOS mini) as quadlets inside its podman machine — not on Dankserver.
 
-- [[ARR-Stack]] — automated media download overview
+| Page | Owns |
-- [[sonarr]] — TV show management
+|---|---|
-- [[radarr]] — movie management
+| [[Alertmanager]] | Alertmanager receives firing rules from Prometheus, groups and dedupes them, and forwards each group as a webhook to… |
-- [[lidarr]] — music management
+| [[Alloy]] | Alloy is Grafana's log and telemetry shipper. |
-- [[readarr]] — ebook/audiobook management
+| [[Anubis]] | Anubis is the bot-challenge proxy that fronts the guarded public vhosts on |
-- [[prowlarr]] — indexer management
+| [[Authentik]] | Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the |
-- [[qbittorrent]] — torrent download client
+| [[authentik-postgres]] | The PostgreSQL 16 database for Authentik — the homelab's SSO / |
+| [[authentik-server]] | The Authentik server — the core, the admin API, and the embedded Caddy outpost. |
+| [[authentik-worker]] | The Authentik worker — background tasks: event processing, provider |
+| [[Caddy]] | Caddy is the single reverse proxy for both hosts. |
+| [[FrankBook-Pro]] | FrankBook-Pro — the Mac mini running the *arr stack + qBittorrent inside a podman machine VM, gluetun-gated |
+| [[Grafana]] | Grafana visualizes the metrics that Prometheus collects from every service. |
+| [[Homepage]] | A password-protected web site on Dankclaw lists the services that are up, with a short description of each and a link… |
+| [[Homepage Design Grilling]] | The decision log for the homepage design. |
+| [[Homepage Status Probes]] | The homepage TUI status page — how probe targets are chosen (host-netns vantage), the three false-down shapes, triage… |
+| [[LAN-DNS]] | The homelab gives every service a name under `franksdank.cloud`. |
+| [[Loki]] | Loki is the log aggregation backend of the observability stack. |
+| [[Node-Exporter]] | Node-Exporter exposes host metrics (CPU, RAM, disk, network) to Prometheus. |
+| [[Observability]] | The Dankclaw observability stack — Prometheus, Grafana, Loki, Alloy, node-exporter and Alertmanager (Signal… |
+| [[Observability Provisioning]] | How Prometheus config/rules and Grafana datasources/dashboards are provisioned as code in serverAppConfigs — the… |
+| [[Pi-hole]] | Pi-hole is the LAN DNS server for the homelab. |
+| [[Prometheus]] | Prometheus collects metrics from every service on Dankclaw by scraping their `/metrics` endpoints, and feeds the… |
 
-## Dankclaw Services
+## Repo architecture and policy
 
-- [[SearXNG]] — privacy metasearch (VPN-routed)
+| Page | Owns |
-- [[SearXNG Providers]] — unblocking Google & other upstream engines behind the VPN
+|---|---|
-- [[Gluetun]] — the VPN tunnel container
+| [[Container Network Namespaces]] | Which local services are addressable from which Podman network namespace on |
-- [[Valkey]] — SearXNG cache and limiter store
+| [[Deploy State Triage]] | Reading and reconciling the deploy-state markers — stale marker vs live container, pending-service lockout,… |
-- [[ComfyUI]] — node-based AI image generation (ROCm GPU, gfx1151 build)
+| [[Deployment Flow]] | Service-scoped Git deployment for the two homelab hosts. |
-- [[ComfyUI Qwen-Image Workflows]] — the two proven Qwen-Image 2.1 workflows + what to add next (LoRAs, Edit, ControlNet, Wan video)
+| [[E2E Suite]] | `tests/e2e/` drives the **live fleet** through a real browser: fresh contexts |
-- [[OpenWebUI]] — chat UI for local LLMs
+| [[Network Architecture]] | One Caddy fronts both hosts. |
-- [[Hermes]] — the Nous Research agent
+| [[Quadlet Lifecycle]] | The full Quadlet discipline on this fleet — ownership auditing, compose-to-Quadlet cutover, drift/crash-loop… |
-- [[Hermes-Dashboard]] — the Hermes web management UI (config, keys, sessions)
+| [[Stale Signal Audit]] | The audit battery for surfaces that rot without failing — unreachable probes, vacuous assertions, invalid checkers,… |
-- [[Hermes-WebUI]] — the Hermex server (loopback API for the iPhone app, SSH-tunneled)
+| [[Storage Policy]] | All persistent data uses **bind mounts**: a host directory mapped directly into the container. |
-- [[Gitea]] — Git hosting (migrated)
+| [[Testing Policy]] | This repository tests its configuration at defined seams. |
-- [[Postgres]] — Gitea database backend
-- [[Seerr]] — unified *arr request manager (guard-chained; fronts FrankBook-Pro's stack)
-- [[Marreta]] — content-cleaning proxy (guard-chained; shares SearXNG's Gluetun egress)
-- [[Open Notebook]] — podcast generator (scripts by frank-zealot, voices by frank-tts); repo-owned config, not deployed
-- [[Frank TTS Engine]] — the arm A TTS engine (`qwentts.cpp` resident `tts-server`, 1.7 B talker) behind frank-tts on a private bridge; deployed on Dankclaw, CPU-only, publishes no port by design
-- [[Frank TTS]] — the 7772 cloned-voice endpoint consumers address: voice naming, pack layout, why the upstream is a container name; deployed, and it is the harness `speech` role
-- [[Voice Packs]] — how a frank-tts voice is built: reference.wav + the pinned engine's speaker artifacts, cut past the 10-second source-material gate
 
-## Infrastructure
+## Operations and incidents
 
-- [[LAN-DNS]] — name resolution, Pi-hole, router DNS hand-off
+| Page | Owns |
-- [[Pi-hole]] — the LAN DNS server (quadlet, records, web UI)
+|---|---|
-- [[Caddy]] — reverse proxy and TLS
+| [[Container Host Reachability]] | The two podman/netavark reachability quirks on the fleet — containers cannot dial unpublished host ports (LAN IP or… |
-- [[Anubis]] — bot challenge proxy fronting the guarded vhosts (chain, policy, gotchas)
+| [[Fleet Audit and Posture]] | Three standing audits — monitoring completeness (what should exist vs what is merely down), vhost proxy health… |
-- [[Authentik]] — the homelab's SSO / OIDC provider (forward_auth)
+| [[Fleet Ops Playbook]] | Incident-shaped fleet procedures — LLM endpoint down with an abandoned GPU holder, scratch vLLM boot windows… |
-- [[authentik-postgres]] — Authentik database service
+| [[Payload and Disk Reclaim]] | Standing procedure for disk pressure on a fleet host — measure all buckets first, classify against the runbook index,… |
-- [[authentik-server]] — Authentik server + Caddy outpost service
+| [[Remote Job Runner]] | Detached long-job execution on fleet hosts with sentinel files (job.sh) — why ssh tail-watchers fail structurally,… |
-- [[authentik-worker]] — Authentik background worker service
+| [[SSH and Remote Access]] | Passwordless SSH bootstrap without sshpass/expect, remote execution against fish login shells, and diagnosing a… |
-- [[Homepage]] — password-protected service index
-- [[Homepage Design Grilling]] — homepage decision log and open questions
-- [[Observability]] — Grafana, Prometheus, Loki, Alloy, node-exporter, Alertmanager: the stack, its quadlet deployment, its exposure, and what is still missing
-- [[Grafana]] — dashboards (datasource and the LLM fleet dashboard provisioned as code since 2026-09-16)
-- [[Prometheus]] — metrics collection (5 alert rules + 1 recording rule, delivered via Alertmanager since 2026-09-17)
-- [[Alertmanager]] — alert grouping and Signal delivery (webhook → host-side relay → signal-cli)
-- [[Node-Exporter]] — host metrics
-- [[Loki]] — log storage
-- [[Alloy]] — log shipping (replaces Promtail)
-- [[FrankBook-Pro]] — Mac mini *arr + qBittorrent inside podman machine (gluetun-gated)
 
-## Research
+## Agent / harness
 
-- [[Service Research Brief]] — consolidated subagent research behind these pages
+| Page | Owns |
-- [[ComfyUI ROCm and Vulkan]] — live-verified GPU path for ComfyUI on Strix Halo
+|---|---|
-- [[Lemonade and sd.cpp]] — the ggml-family image-generation alternative
+| [[Agent Docs and Edits]] | Two self-inflicted damage classes — structurally broken markdown after parallel agent edits, and botched hashline… |
-- [[Llama Cpp Runtimes]] — mainline vs charlie12345/ROCmFPX vs ik_llama.cpp: quantization + server fit
+| [[Command Center Dotfiles]] | The ~/.omp repo is itself the deployment — stow-equivalent packages, full-replica rsync transport, the pre-overwrite… |
-- [[Llama Cpp Code Diff]] — code-level quantization diff of the three llama.cpp runtimes
+| [[OMP Harness]] | The omp agent harness as configured on this fleet — compaction chain arithmetic, per-model sampler routing via… |
-- [[SGLang]] — the structured-generation serving engine (llama.cpp/vLLM comparison + our fit)
-- [[Backblaze Integration]] — off-site backup plan (future)
-- [[Jellyfin]] — media server research brief
-- [[ARR-Stack]] — automated download stack research brief
-- [[Local TTS and Voice Cloning]] — creating a local voice from samples, the 2026 cloning landscape, the full audio stack, and a recommended homelab deployment
-- [[AI Job Search and Resume Tools]] — self-hosted AI job-search + resume-formatting apps, local-LLM fit, recommended stack
-- [[Most Popular Self-Hosted AI Apps]] — the 15 most popular self-hosted AI apps of Jun–Sep 2026, homelab fit, adoption best practices
 
-## Skill Distillation (2026-10-05)
+## Research and methodology
 
-The managed skill library distilled into the vault, one reference page per
+| Page | Owns |
-skill-cluster. See [log.md](log.md) for the wave plan and results.
+|---|---|
+| [[AI Job Search and Resume Tools]] | Self-hosted AI job-search and resume-formatting applications: the 2026 landscape, local-LLM fit, and a recommended… |
+| [[Backblaze Integration]] | Off-site backup plan for the bind-mounted data on both hosts. |
+| [[ComfyUI ROCm and Vulkan]] | Research into whether ComfyUI can use GPU acceleration — not just CPU — on the AMD Strix Halo APU (gfx1151) in… |
+| [[Lemonade and sd.cpp]] | Why Lemonade Server can do image generation on Strix Halo when ComfyUI cannot. |
+| [[Llama Cpp Code Diff]] | Code-level diff of mainline llama.cpp, charlie12345/ROCmFPX, and ik_llama.cpp with a focus on quantization code, ggml… |
+| [[Llama Cpp Runtimes]] | Compared mainline llama.cpp, charlie12345/ROCmFPX, and ik_llama.cpp — positioning, quantization formats, code-level… |
+| [[Local TTS and Voice Cloning]] | How to create a new local TTS voice from sample files in oh-my-pi, the 2026 open-weight voice-cloning landscape, the… |
+| [[Most Popular Self-Hosted AI Apps]] | The 15 most popular self-hosted AI applications of June–September 2026, ranked by GitHub stars, with homelab fit and… |
+| [[PICO-8 Asset Standards]] | Production-grade standards for PICO-8 asset sheets — palette discipline, single perspective and light direction,… |
+| [[Service Research Brief]] | Consolidated research from the `frank-zealot` research subagents (2026-08-14) plus the pi agent knowledge base. |
+| [[SGLang]] | What SGLang is, how it works (RadixAttention, paged KV, overlap scheduler, structured output), how it compares to… |
 
-**Operations:**
+## LLM technique and hardware references
 
-- [[Remote Job Runner]] — detached long-job execution with sentinel files
+| Page | Owns |
-- [[Payload and Disk Reclaim]] — disk reclaim: measure, classify, prove,
+|---|---|
-  stage, ask; plus purge-and-refresh
+| [[Dankclaw Improvements]] | This page collects concrete, sourced recommendations to improve Dankclaw |
-- [[Container Host Reachability]] — netavark/pasta reachability quirks and
+| [[Dankclaw System Specs]] | Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference. |
-  the pasta port gate
+| [[dankclaw-iommu-translation-off]] | Dankclaw boots iommu=pt since 2026-10-18 — identity DMA with the IOMMU armed, which restored the XDNA2 NPU at a… |
-- [[Fleet Ops Playbook]] — endpoint-down GPU-holder reclaim, scratch-boot
+| [[Dankserver System Specs]] | Dankserver hardware and OS state since the 2026-09-20 board swap (ASUS ProArt B850-Creator WiFi NEO, 2x AMD Radeon AI… |
-  stop-first, pre-commit iteration, incident-window reconstruction, WoL
+| [[dankserver-frontier-moe-feasibility]] | VERIFIED 2026-09-28 by measurement and source reading. Compares DeepSeek-V4.1-Flash, MiMo-V2.6-Flash-MOPD and… |
-- [[Fleet Audit and Posture]] — monitoring gaps, vhost probes, security
+| [[dankserver-vram-growth]] | Investigation of the recurring cuBLAS crash on frank-zealot. What grows, what does not, and what to do next. |
-  posture, whole-fleet audit, bulk redaction
+| [[dealign-ornith-crack]] | dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with… |
-- [[SSH and Remote Access]] — passwordless bootstrap, fish login shells,
+| [[deployment]] | Scoped deployment and update process for the three inference services (frank-archon, frank-dragoon, frank-zealot). |
-  GUI-hang diagnosis over ssh
+| [[DFlash DSpark Speculative Decoding]] | DFlash and DSpark are the two newest speculative decoding methods. |
+| [[Dual-R9700-Cooperation]] | Why frank-medium's two Radeon AI PRO R9700s could not run tensor-parallel on the old B650-E board (PCIe atomic… |
+| [[FastFlowLM NPU Inference]] | FastFlowLM (FLM), the AMD NPU-first LLM runtime for Ryzen AI XDNA2 chips. Fit analysis for the 50 TOPS NPU on… |
+| [[FreeToken Builds]] | Building FreeToken (the official franksdank fork of FreeTok Tokenworks' freetoken vLLM-MTP runtime) on Dankclaw — the… |
+| [[freetoken-benchmark-sweep]] | KV pool vs decode speed sweep across bs1/bs3/bs4/bs6 on Dankserver with Qwen3.6-35B-A3B-FP8 via FreeToken v0.1.2. |
+| [[GGUF Compatibility]] | How to answer "will engine X load this GGUF?" from the file's header alone, why a custom tensor type forces the… |
+| [[Heterogeneous Split Triage]] | Triage before building when someone asks to split a model across unequal accelerators on one host (24 GB GDDR6X +… |
+| [[Hybrid CPU GPU Processing Dankserver]] | Dankserver uses NVIDIA GPU with system RAM for hybrid processing. |
+| [[ling-3.0-flash-trial]] | `bailingmoe3-support` (commit `db4480bc8`, build b10411) at `/home/dank/llama.cpp-ling`. |
+| [[ling-3.0-flash-vllm-plan]] | This page was the plan for a vLLM-ROCm deployment of `jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native` (INT4 W4A16, ~62… |
+| [[Llama Cpp Multi-Device Splits]] | What llama.cpp can and cannot express across devices, the no-collective layer-split path, split balancing by… |
+| [[Llama Cpp Serving Sizing]] | Slot-pool sizing against a headroom band, per-device VRAM accounting, prompt-cache/state sizing and the QSA… |
+| [[LLM Bench Windows and Measurement]] | Stopping-and-restoring benchmark windows on the fleet LLM hosts (llama-bench sweeps, HTTP param benches, multi-arm… |
+| [[LLM Performance Diagnosis]] | Probing why a served LLM is slow — the flat-depth probe, the expert-offload duty-cycle arithmetic, kernel-time… |
+| [[Model Swap and Rollback]] | The full model-swap discipline for the LLM fleet — fit preflight, fetch and payload verification, runtime build/flag… |
+| [[PCIe Link and Atomics]] | Link probing (idle downshift, path ceilings, CUDA/HIP bandwidth probes), the streaming-vs-compute ceiling arithmetic,… |
+| [[Quantization by Hardware Architecture]] | This page maps LLM quantization formats to the GPU and CPU architectures that support them. |
+| [[Quantization Production]] | — |
+| [[qwen3.8-strix-optimizations]] | This page answers one question: **how do we make Qwen3.8-27B serve faster on |
+| [[RAM CPU Offloading in VLLM]] | vLLM uses CPU RAM and swap space to extend GPU memory capacity. |
+| [[RDNA4 Container Serving]] | Bringing a model larger than VRAM onto Dankserver's 2x R9700 (gfx1201) with the tcclaviger vLLM stack — quant-class… |
+| [[RDNA4 MoE Expert Offload VRAM Budget]] | The expert-offload plan arithmetic on the tcclaviger fork — which terms a deploy can move, the util law, the… |
+| [[Running Qwen3.6 27B FP8 on Dankclaw]] | Qwen3.6-27B is a dense vision-language model from the Qwen3.6 series. |
+| [[Running Qwen3.6 35B A3B FP8 on Dankserver]] | Qwen3.6-35B-A3B is a Mixture-of-Experts vision-language model from the Qwen3.6 series. |
+| [[Shared Memory Architecture Dankclaw]] | Dankclaw uses AMD Strix Halo APU with unified memory architecture. |
+| [[swift15-campaign-worklog]] | — |
+| [[Tcclaviger Fork Ops]] | Operating the tcclaviger RDNA4 vLLM fork on dankserver (shape G era, rollback target since the 2026-10-05 FreeToken… |
+| [[ubatch-benchmark-large]] | Standardized ubatch sweep for the active kingjones Qwen3.8 ROCmFPX deployment. |
+| [[vLLM Boot Forensics and Config Interrogation]] | Capturing a vLLM container boot failure's real root cause from journald (the container log is already gone),… |
+| [[VLLM Index]] | This index contains all documentation for vLLM deployment on Dankserver and Dankclaw. |
+| [[VLLM Overview]] | vLLM is a fast and easy library for large language model inference and serving. |
+| [[vLLM Porting and Compile Triage]] | Forward-porting a model architecture to a new vLLM major (additive-only discipline, gap enumeration, shims that… |
+| [[vllm_research_findings]] | vLLM v0.26.0 (latest stable, released July 25, 2026) supports DFlash speculative decoding with the `--spec-method… |
+| [[x4-pair-experiment]] | What mainline llama.cpp can and cannot do with a 3090 + Strix Halo combination, measured 2026-09-16 on Dankclaw — the… |
 
-**Architecture:**
+## Pipeline Parallelism (series)
 
-- [[Stale Signal Audit]] — classifying green-but-dead and down-but-fine
+| Page | Owns |
-  signals
+|---|---|
-- [[Quadlet Lifecycle]] — quadlet units, drift, crash-loops, compose
+| [[Ethernet Bandwidth Math]] | Bytes-per-token math for pipeline vs tensor parallelism, feasibility tables for 1/2.5/10 GbE, and real multi-node… |
-  cutover
+| [[Expected Performance]] | Estimated tokens per second for a 25-40B model split across two nodes over 1/2.5/10 GbE, with real benchmark anchors. |
-- [[Deploy State Triage]] — revision markers, pending lockouts, deploy
+| [[llama.cpp RPC]] | llama.cpp distributed inference via ggml-rpc-server and the --rpc flag: architecture, CLI, versions, limitations, and… |
-  interruptions, shared-checkout hygiene
+| [[Mixed-Vendor Experiments]] | Real-world NVIDIA + AMD Strix Halo multi-node experiments: what works (llama.cpp RPC, Vulkan split), what breaks… |
+| [[Qwen3.6 27B Spec Methods]] | Qwen3.6-27B Q8 GGUF variants and llama.cpp speculative decoding (draft-mtp, draft-dflash, ngram-mod): flags, RPC… |
+| [[Qwen3.8 PP Deployment]] | Split Qwen3.8-27B across the Strix Halo iGPU (head) and the RTX |
+| [[Sample Script Set]] | An educational two-node llama.cpp RPC split of Qwen3.6-27B-Q8_K_P over ethernet. For reading only. Do not run or… |
+| [[vLLM Pipeline Parallelism]] | vLLM multi-node pipeline parallelism: flags, Ray/mp backends, interconnect requirements, PP bubble, and PP +… |
 
-**Infrastructure:**
+## Uncategorised
 
-- [[Observability Provisioning]] — Prometheus/Grafana config as code
+| Page | Owns |
-- [[Homepage Status Probes]] — the status page's checker, vantage rules,
+|---|---|
-  and triage
+| [[log]] | — |
-
-**LLM cross-cutting (full index: [LLM/index.md](LLM/index.md)):**
-
-- [[GGUF Compatibility]] · [[Llama Cpp Serving Sizing]] ·
-  [[Llama Cpp Multi-Device Splits]] · [[PCIe Link and Atomics]] ·
-  [[RDNA4 Container Serving]] · [[RDNA4 MoE Expert Offload VRAM Budget]] ·
-  [[vLLM Boot Forensics and Config Interrogation]] ·
-  [[vLLM Porting and Compile Triage]] · [[LLM Bench Windows and Measurement]] ·
-  [[LLM Performance Diagnosis]] · [[Model Swap and Rollback]] ·
-  [[Tcclaviger Fork Ops]] · [[Heterogeneous Split Triage]] ·
-  [[FreeToken Builds]] · [[Quantization Production]]
-
-**Agent:**
-
-- [[OMP Harness]] — compaction arithmetic, per-model samplers, preamble
-  audits
-- [[Agent Docs and Edits]] — doc-structure audits, hashline edit recovery
-- [[Command Center Dotfiles]] — the ~/.omp repo-as-deployment
-
-**Desktop / Research:**
-
-- [[Noctalia and Niri]] — the desktop shell, palette recolor, template
-  rendering, rotation pinning, iCX3 RGB
-- [[PICO-8 Asset Standards]] — cohesive PICO-8 asset sheets
-
-## LLM Fleet
-
-The LLM inference fleet on the two hosts. Full index:
-[serverAppConfigs-docs/LLM/index.md](LLM/index.md).
-
-### Deployment and Topology
-
-- [[deployment]] — unified deploy and update process for both servers
-- [[frank-archon-radiance-qwen38-iq4r]] — **ACTIVE frank-archon runbook
-  (2026-10-15)**: the PUBLISHER's container of the official
-  Qwen3.8-Flash-Next (`qwen3.8-next-flash-fp8-iq4r-moe.rad`, 116,322 MiB,
-  w4nl 4-bit-codebook experts + int8 trunk, Hub-sha256-verified at fetch)
-  swapped in as a MODEL_FILE-only change — the 2026-10-13 shape (slab
-  25600 / fp8-KV 5120, 3 × 200,000, host pool 44,928, MTP depth 2) carries
-  verbatim; rollback = revert the swap commit (the w4dn8 `.rad` stays on
-  disk; the fp8-native `.rad` was deleted 2026-10-15, rebuildable)
-- [[frank-archon-radiance-swift15-w4dn8]] — RETIRED 2026-10-08 → 10-15 frank-archon runbook:
-  Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 `.rad` (radiance
-  1.0.13), gated on KLD ≤ 0.03 vs the BF16 container and decode ≥ 30.8 t/s;
-  activates by flipping `MODEL_FILE` in `scripts/frank-archon.env`, rollback
-  = revert that line (the FP8-native `.rad` stays on disk)
-- [[frank-archon-radiance-qwen38-native-fp8]] — the RETIRED 2026-10-05 → 10-08 frank-archon
-  runbook (2026-10-05): the StillDeadcode radiance engine (digest-pinned
-  287 MiB image, libr4d gfx1201 kernels) serving the OFFICIAL
-  `Qwen/Qwen3.8-Flash-Next-FP8` at native block-FP8 (no-recipe convert
-  after the descriptor-staging drop) on BOTH R9700s TP=2 `wht6`, experts
-  in a 40 GiB/rank pinned host pool, PLE 47.7 GiB pinned to RAM, fp8 KV
-  7.32 KiB/token, MTP depth 3, 2 × 200,000 first boot; rollback is the
-  FreeToken one-card era (tag `frank-archon-freetoken-pr587-20261005`),
-  then the tcclaviger shape-G era (tag `frank-archon-vllm-shapeG-20261003`)
-- [[swift15-campaign-worklog]] — ACTIVE campaign log: the Swift1.5 →
-  w4dn8 radiance convert and frank-archon swap in flight (branch
-  `swift15-radiance-w4dn8`); the cutover runbook replaces it at completion.
-- [[frank-archon-freetoken-pr587]] — RETIRED 2026-10-05 (superseded the
-  same day by the radiance cutover above; kept as the rollback runbook):
-  FreeToken built at PR #587 head `77a7201` on the host into
-  `frank-archon-freetoken:pr587-77a7201`, ONE R9700 with experts in pinned
-  host RAM behind a 0.10 GPU LRU, 2 slots, parsers AUTO
-- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
-- [[frank-archon-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
-  back; rollback target): the R9V radiance-vLLM fork (docker-pinned bundle
-  `2dac17a2`, docker→podman shim) serving the unsloth
-  Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 shards) + blockfp8 MTP4 +
-  BF16 vision on Dankserver's both R9700s — TP=2 tiered experts (80-slot
-  LRU cache), PLE pinned 54.4 GB in host RAM, CED off, 131,072 ctx single
-  stream, Qwen-Sharp template, root unit with LimitMEMLOCK, the on-host
-  qualified state, the cutover, and the rollback
-- [[frank-archon-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
-  (rollback target): the R9V radiance-vLLM fork serving the abliterated
-  Dyluhn Qwen3.8-Flash-Next-Uncensored IQ4_XS (92.39 GiB) + MTP4 + F16
-  vision — TP=2 tiered experts with a UVA pinned host copy, PLE on
-  ssd-mmap, CED on, 196,608 ctx; state dir + frozen config + payload
-  intact on disk
-- [[frank-archon-llamacpp-qwen38-flash-next-q8]] — RETIRED 2026-09-27
-  (rollback target): unsloth llama.cpp (b11030-mix prebuilt, ROCm
-  gfx120X, root unit) serving unsloth Qwen3.8-Flash-Next at Q8_0 (175.3
-  GiB) on Dankserver's both R9700s — the shared Q8_0 MTP head chained
-  behind ngram-mod, BF16 vision, the experts and the 51B n-gram table
-  computed in host RAM, the placement law, the cutover, and the rollback
-- [[frank-archon-lucebox-ds4]] — RETIRED 2026-09-26 (same-day swap;
-  rollback target): the Lucebox runtime serving
-  DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, **40**-layer MoE, **384**
-  routed experts top-6, 189 GiB of embedded Engram) on Dankserver's R9700
-  with the routed experts streamed from the mmapped page cache — held :7770
-  for hours
-- [[dankserver-frontier-moe-feasibility]] — VERIFIED 2026-09-28: can
-  DeepSeek-V4.1-Flash (552B), MiMo-V2.6-Flash-MOPD (309B) or GLM-5.3-Flash
-  (320B) run fast on the 2x R9700 box, and can Affinity / vllm-radiance /
-  FreeToken-AMD be modified for them? The capacity arithmetic (475 GiB
-  payload, 189 GiB of it Engram, ~2.4 GB of cold expert weights per token
-  across PCIe = a ~23 tok/s ceiling) plus the Affinity V4-Flash-to-V4.1 port
-  assessment — reusable expert tiering, four architecture gaps, one a
-  silent-wrongness trap on the Engram key
-- [[dankserver-qwen38-fp8-expert-offload-tuning]] — VERIFIED 2026-09-29:
-  the flag-level decode-tuning campaign on the ad hoc qwen-fp8 container
-  (tcclaviger RDNA4 fork, orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8,
-  TP=2 across both R9700s). **>35 tok/s MET: 37.44-40.57 tok/s with
-  `--expert-prefetch` DELETED** — telemetry measured the prefetcher at
-  36% precision, wasting ~half the gather traffic (the LRU itself runs at
-  82.45% hit); the prefetch-on incumbent 26.8 survived every other
-  challenger (geometry, spec depth, TunableOp-corrupts, batch knobs,
-  concurrency, ngram-fork-bug); the residual gap to ~60 tok/s is the
-  fork's gather read path (6.5-9.5 GB/s effective vs 22.7 at the same
-  slab size)
-- [[frank-archon-glm53-flash-w4a16]] — VERIFIED 2026-09-30: the ACTIVE
-  frank-archon runbook. The cutover to Intel/GLM-5.3-Flash-W4A16-AutoRound
-  (symmetric int4 g128) on the tcclaviger fork, and the reason the int4
-  AutoRound payload is the ONLY one that serves here (its `auto-round` quant
-  method routes to the W4A16 offload kernel; a compressed-tensors W4A16
-  checkpoint of the same model reaches TritonWNA16Experts, which has no
-  offload recipe). **Measured decode 9.9-10.0 tok/s / prefill 918 tok/s; the
-  35 tok/s target is NOT met** — the step is bounded by the fork's gather
-  read path (~8.2 GB/s vs a 22.74 GB/s link) with the cache at a 0.53-0.66
-  hit rate, and MTP4 was measured and rejected (zero decode gain, prefill
-  918 -> 255). Carries the fork's KV-cap bug, the fp8-KV/gfx12 trap, and the
-  `glm47` tool/reasoning parser wiring
-- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26:
-  vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D one-shot
-  allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4
-  W4A8, DFlash2 drafter, unquantized bf16 KV since 2026-09-21) on Dankserver's
-  2x Radeon AI PRO R9700 behind the :7770 endpoint — the quantization-aware
-  entrypoint, the three-arm battery that chose TP=2, verified capabilities,
-  and the record of the 2026-09-16 two-replica era
-- [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium:
-  the streamlining toolchain (hf-manifest.py, hf-fetch.sh,
-  swap-frank-medium.sh), the quantization-aware entrypoint contract, the
-  quadlet edit, the deploy, the capability battery, and the rollback path
-- [[frank-medium-llamacpp-dflash2]] — RETIRED 2026-09-16 → 2026-09-20:
-  mainline llama.cpp (gfx1201 HIP build) serving orcarouter Qwen3.8-27B at
-  Q8_0 with the DFlash2 drafter as ONE layer-split process (32:33, two
-  163,840-token slots); holds the batch/ubatch optimum curve and the
-  per-layer split arithmetic the rollback recipe points back to
-- [[Dual-R9700-Cooperation]] — whether the two R9700s can run ONE model: the
-  RCCL/PCIe-atomics evidence on the old TUF GAMING B650-E (which closed tensor
-  parallelism too), what the 2026-09-20 ProArt B850-Creator x8/x8 swap changed
-  — both chains `Routing+`, both endpoints `ReqEn+` — and the measured
-  three-way verdict: radiance TP=2 won and is the deployed shape
-- [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era):
-  charlie12345/ROCmFPX serving kingjones777/
-  Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the runtime it
-  replaced
-- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep (bs1/bs3/bs4/bs6, 8k–700k KV) on the retired Dankserver RTX 3090
-- [[kingjones-strix-mtp]] — the Qwen3.8-27B-Q6_0 ROCmFPX AGENT era with the
-  standalone MTP draft (Dankclaw, 2026-08-25 → 2026-08-29; historical)
-- [[agentionai-fast-imatrix]] — the retired frank-dragoon runbook:
-  agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB,
-  Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30;
-  rollback record)
-- [[frank-dragoon-halogen27b-dflash2]] — RETIRED 2026-10-05 → 10-06 frank-dragoon
-  runbook (root systemd unit + podman, port 7770): the digest-pinned
-  ghcr.io/peonist-ai/halogen image serving peonist-ai/halogen-qwen3.8-27b
-  (qwen3.8-27b-p1w4d-d2.hgn, 33.4 GiB — a .hgn that loads in no other
-  engine) with the DFlash2 drafter on ONE slot of 200,700 ctx (slots>1
-  disables speculation — engine law); TEXT-ONLY and censored base, both
-  accepted tradeoffs; no server-side sampler, so the thinking recipe rides
-  the wire
-- [[frank-dragoon-flash-server-v2-native]] — **ACTIVE frank-dragoon runbook
-  since 2026-10-17**: halogen-flash-server 0.16.2 serving the **NATIVE v2
-  abliterated** conversion of orcarouter's pinned BF16
-  (`qwen38-flash-next-v2-abliterated.hgn`, 62.1 GiB) at ctx **200,704** over a
-  **401,408**-position pool (`HALOGEN_KV_POOL_POSITIONS=401408`, 11.0 GiB
-  reserved — one pool behind both slots, raised to 2 × ctx and deployed
-  2026-10-18 so TWO maximum-length
-  requests coexist) and the stock peonist n-gram/vision companions **hardlinked**
-  in — engine-tool certified (NIAH 18/18, PPL 3.968, mean KL 0.155); MTP
-  always-on, vision ON, no server-side sampler change
-- [[frank-dragoon-flash-server-v2-swift15]] — RETIRED 2026-10-06 → 10-17
-  frank-dragoon runbook (rollback target — payload + fetch script on disk):
-  halogen-flash-server 0.16.2 serving the abliterated **Swift1.5 finetune**
-  of the Qwen3.8-Flash-Next v2 `.hgn`, two slots over a 524,288-position
-  pool; owns the 400-zone law and the sampler/prompt-cache contract
-- [[frank-dragoon-rocmfpx-q6agent-uncensored-mtp]] — RETIRED 2026-10-03 → 10-05
-  (rollback target; payload, fork build and build script on disk): the pinned
-  kingjones30/ROCmFPX fork (scripts/build-rocmfpx.sh) serving kingjones777/
-  Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF
-  **Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT** — ftype 114, 339 tensors of
-  GGML_TYPE_Q6_0_ROCMFPX + 159 type-103, a format no other loader on the
-  host reads — with the repo's Q4_0 MTP head drafting 4 tokens (27.6-28.1
-  t/s measured, 86.8 % acceptance; the ngram-mod chain measured and
-  rejected) and its BF16 projector, one session at the native 262,144 window
-- [[frank-dragoon-rocmfpx-q8agent-mtp]] — RETIRED 2026-10-02 → 10-03 (rollback
-  target, aligned Q8 payload still on disk): the same fork serving the
-  ALIGNED Q8_0_ROCMFPX_AGENT at 17.0 t/s — the +123 % MTP A/B, the traps
-- [[frank-dragoon-gufo-qwen38-27b]] — RETIRED 2026-10-01 → 10-02 (history;
-  the payload dirs were removed from disk 2026-10-03, `scripts/build-gufo.sh`
-  remains): gufo v0.1.1
-  built from source serving Qwen3.8-27B UD-Q4_K_XL + the z-lab DFlash2
-  drafter on the gfx1151 iGPU — the two from-source build workarounds, the
-  template-hash allowlist, the measured prefill/decode trade
-- [[frank-dragoon-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (same day;
-  replaced by the gufo rollback, then by
-  [[frank-dragoon-rocmfpx-q8agent-mtp]]): the digest-pinned
-  kyuz0/vllm-therock-gfx1151 image (vLLM 0.30, TheRock ROCm 10) serving
-  XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16 on the gfx1151 iGPU,
-  4 x 262,144 ctx, with CROSS-MODEL MTP spec decode from Qwen/Qwen3.5-9B —
-  the roof math (13.1 t/s base is ~94% of LPDDR5x BF16), the measured
-  MTP-vs-DFlash table (19.8 vs 17.4 t/s), the greedy-text numerics trap,
-  the `reasoning` field change, troubleshooting
-- [[frank-dragoon-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
-  [[frank-dragoon-gufo-qwen38-27b]]; rollback target): stock llama.cpp
-  (unsloth b11030, ROCm gfx1151) serving
-  unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter
-  (depth 4) on the gfx1151 iGPU — one 262,144-token conversation, q8_0 KV
-  (the build lacks fp8), text-only, Qwen-Sharp v22.4.0 template, the
-  114 GiB GTT memory model, client contract
-- [[frank-dragoon-kairic-edge-qwen38-27b]] — RETIRED 2026-09-21 (replaced by
-  [[frank-dragoon-unsloth-qwen38-27b-q8xl]]; same-day swap): the Kairic Edge
-  runtime (ciru-ai/ROCmFPX kairic-edge-qwen38-27b-v1.2, TheRock 7.15.0,
-  Composable Kernel) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (a 27B
-  dense model, main GGUF + FFN/GDN/GDN-Output .pfs sidecars) on the gfx1151
-  iGPU — one 262,144-token conversation, native MTP depth 4 + n-gram
-  drafting, compatibility mode, the 114 GiB GTT memory model, client contract
-- [[frank-dragoon-unsloth-qwen38-flash-next]] — RETIRED 2026-09-21 (replaced by
-  [[frank-dragoon-kairic-edge-qwen38-27b]]; same-day swap): the unsloth
-  llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving
-  unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (a 125B MoE / 6B active) on the
-  gfx1151 iGPU — one 200,000-token conversation, MTP depth 2 via the
-  explicit -md shared-Q8_0 head, the 114 GiB GTT memory model, client contract
-- [[frank-dragoon-ciru-orca]] — RETIRED 2026-09-21 (replaced by
-  [[frank-dragoon-unsloth-qwen38-flash-next]]; rollback target, 2026-09-19 →
-  09-21): the CIRU v4.4.1 runtime (private host build, QSA hotfix) serving
-  jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 iGPU — one
-  262,144-token conversation, MTP depth 4, 4096 MiB PLE cache, runtime-root
-  gate, client contract
-- [[halogen-flash-server]] — RETIRED 2026-09-19 (replaced by
-  [[frank-dragoon-ciru-orca]]; rollback target, 2026-09-16 → 09-19):
-  peonist-ai/halogen-flash-server 0.5.8 serving peonist-ai/halogen-qwen3.8-
-  flash-next (w4b `.hgn` + quality overlay) on the gfx1151 iGPU — two 200k
-  conversations over one 400,384-position pool, deployed flags, memory model,
-  client contract
-- [[frank-dragoon-ik-uncensored]] — the RETIRED 2026-09-16 frank-dragoon runbook
-  and rollback target (2026-09-15 → 09-16): ik_llama.cpp serving
-  mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090 +
-  Strix Halo hybrid — its measured envelope is why the fleet went back
-- [[dankclaw-iommu-translation-off]] — Dankclaw's IOMMU boot line: `iommu=pt`
-  since 2026-10-18 (measured cost of identity DMA: nothing — +2.0 % prefill), the
-  `amd_iommu=off` era it replaced, and the tracked file + installer that own it
-- [[kingjones-strix-rocmfp4]] — the PRIOR frank-dragoon runbook
-  (2026-08-30 → 2026-09-11) and the rollback target:
-  kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF on the card's
-  pinned kingjones30/ROCmFPX build
-- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
-- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on Dankclaw
-- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
-- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
-- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
-- [[frank-zealot-llama-prism-bonsai9-mtp]] — the **ACTIVE** frank-zealot
-  runbook since 2026-10-18: the PrismML vendor llama.cpp build 10777 (the only
-  loader for the custom PQ2_0 tensor type / `prism.hadamard.*`, and the floor
-  for MTP beside a quantized V cache) serving
-  prism-ml/Ternary-Bonsai-2-9B-gguf **PQ2_0 with the MTP head** on the RTX 3090
-  — TWO 262,144-token conversations (`--ctx-size 524288 --parallel 2`), exact
-  f16 K + f16 V = **16,384 MiB** over 8 full-attention layers (32,768 B/token;
-  the per-slot window is UNCHANGED at 262,144 — what halved is the POOL,
-  because exact f16 over the OLD 786,432 pool bills **24,576 MiB** against the
-  **24,123** the driver leaves addressable; the old "26,112 vs 13,056 MiB" note
-  doubled the q8_0 figure with its 1/32 scale overhead included, so f16/q8_0 is
-  1.882×, not 2×), MTP **off by measurement** (the draft context bills 2,048
-  MiB of f16 KV over the whole pool, and exact KV leaves no room for one),
-  vision through the Q8_0 mmproj; boot 17,876 used / 6,249 free (the q8_0 era)
-- [[frank-zealot-llama-prism-bonsai2-pq20]] — RETIRED 2026-10-17 → 10-18
-  (rollback target, payload + build 10754 on disk): Ternary-Bonsai-2-**27B**
-  PQ2_0 at TWO 200,704-token conversations, q8_0 K + q8_0 V, MTP **impossible**
-  (no head in that export), vision through the Q8_0 mmproj; boot 22,664 / 1,461
-- [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — RETIRED 2026-10-17 (rollback
-  target, payload on disk): llama.cpp master (CUDA sm_86) serving
-  HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** on the RTX 3090 —
-  TWO 200,704-token conversations, exact f16 K + f16 V, MTP **impossible**
-  (the file carries no `nextn`/`mtp` tensors), vision through the publisher's
-  BF16 mmproj; priced from the MiMo Q6_K boots (same qwen35 geometry)
-- [[frank-zealot-llama-gemma4-q4km-200704]] — RETIRED 2026-10-07 (same day;
-  payload on disk as rollback): llama.cpp master (CUDA sm_86) serving
-  HauhauCS Gemma4-26B-A4B QAT Uncensored **Q4_K_M** on the RTX 3090 — ONE
-  200,704-token conversation, f16 K + q8_0 V, MTP **disabled** (the
-  `gemma4-assistant` drafter costs ~1.9 GiB: boot was 23,806 used/319 free
-  with it, 21,890/2,235 without), vision through the publisher's mmproj
-- [[frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688]] — RETIRED 2026-10-07
-  (superseded by gemma4; payload on disk as the rollback): the
-  frank-zealot runbook 2026-10-05 (evening) → 2026-10-07
-  (**UNCECERSED RCO-IQ3_XXS
-  with MTP back on since 2026-10-06 evening; renamed from frank-small and
-  reshaped to TWO 114,688-token conversations at Frank's 112k cap with
-  the ubatch ladder climbed to 2048 late 2026-10-06**): llama.cpp master (CUDA
-  sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS
-  (the MTP head in the file is drafted) on the RTX 3090 — 1,005 MiB free
-  at boot (763 settled after a vision probe), q8_0 K + q8_0 V, vision
-  through the publisher's own projector, ~52–58 t/s recipe decode with MTP
-- [[frank-zealot-llama-mimo-q6k-2x200704]] — RETIRED 2026-10-05 (evening,
-  superseded by swift15): llama.cpp master serving
-  bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** — TWO conversations at
-  200,704 tokens each with fully unquantized f16 KV and mmproj vision;
-  94.4 tok/s decode (the Q6_K rung costs a measured −10 % vs Q5_K_M —
-  weight-bandwidth), batch 4096 / ubatch 2048 (+4.6 % prefill, measured
-  2026-10-05), 2,279 MiB free, and the ≥256 px image-probe trap (tiny
-  images answer "video")
-- [[frank-zealot-llama-mimo-q5km]] — RETIRED 2026-10-02 → 10-03 (rollback
-  target, payload on disk; owns the measured boot-pair table): same runtime
-  and shape one rung lower — 104.8 tok/s, 3,571 MiB true free
-- [[frank-zealot-llama-mimo-q6k]] — RETIRED 2026-10-01 (same day; that same
-  Q6_K rung returned ACTIVE 2026-10-03 at the new shape, see
-  [[frank-zealot-llama-mimo-q6k-2x200704]]): the first llama.cpp + GGUF shape
-  for this endpoint, Q6_K at 2 x 262,144 / f16-K + q8_0-V, 20.9 GiB measured
-- [[frank-zealot-sglang-mimo-v26]] — RETIRED 2026-10-01 (rollback target):
-  SGLang in a self-contained python3.12 venv serving
-  letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5
-  hybrid, INT8 W8A16, in-checkpoint vision, NO MTP weights shipped) on
-  Dankclaw's RTX 3090, port 7771 — fp8 KV, pinned 606,208-token pool,
-  2 x 262,144 slots, bf16 mamba states, eager prefill
-- [[frank-zealot-gemma4-12b]] — RETIRED 2026-10-01 (rollback target):
-  mainline llama.cpp serving
-  HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
-  (MTP assistant sidecar + ngram-mod, q8_0 KV) on
-  Dankclaw's RTX 3090, port 7771
-- [[frank-zealot-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
-  rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
-  Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-
-  token pool, 2 concurrent full-context sessions, native text+image+audio
-- [[frank-zealot-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target):
-  the prior frank-zealot runbook (2026-09-16 → 09-17), ISTA-DASLab
-  Qwen3.8-27B-GSQ-RCO IQ3_S with own in-file MTP head + BF16 projector
-- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for the old frank-zealot (q8_0 + Hadamard KV cache, split-mode none; 2026-08-26)
-- [[freetoken-frank-small]] — RETIRED 2026-09-15: the FreeToken/CUDA frank-small runbook, retired with the RTX 3090
-- [[kairic-edge]] — the retired Kairic Edge IU4 experiment (2026-08-22)
-- [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment (2026-08-21)
-- [[Dankserver System Specs]] — the 2x R9700 box hardware (frank-medium)
-- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-dragoon)
-- [[Dankclaw Improvements]] — host-level changes on the Strix Halo box
-- [[Swap Configuration]] — the 64 GiB `/swap.img` host swapfile (raised from
-  8 GiB 2026-09-12): fstab-driven `swap.img.swap` unit, the safe resize
-  ordering that drains the old file into fresh swap instead of into a host
-  whose `MemAvailable` lies by the size of the model, and the swappiness caveat
-- [[frank-npu]] — the RETIRED FLM NPU endpoint on Dankclaw (2026-09-11):
-  Gemma4 E4B vision+audio pack on the XDNA2 NPU, port 7771; kept as the
-  retirement record, including the IOMMU/SVA err=-19 root cause and
-  where every dependent route moved
-
-### Benchmarks and Sizing
-
-- [[frank-dragoon-benchmark]] — backend/spec benchmark behind the old head
-- [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results
-- [[frank-zealot-benchmark]] — the small-model benchmark suite
-- [[frank-medium-batch-ubatch]] — the 2026-09-16 batch/ubatch sweep on the 2x
-  R9700 layer split: prefill PEAKS at ubatch 512 and falls on both sides, the
-  logical batch buys nothing measurable, decode never moves, and the ubatch's
-  uneven VRAM give-back (1.30 GiB on card 0 against 2.41 GiB on card 1) moved
-  the tensor split from 36:29 to 35:30
-- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware notes
-- [[Running Qwen3.6 27B FP8 on Dankclaw]] — Qwen3.6-27B sizing on the iGPU
-- [[qwen3.8-strix-optimizations]] — Strix Halo tuning notes
-- [[x4-pair-experiment]] — what a 3090 + Strix Halo combination can and cannot
-  do in mainline llama.cpp (measured 2026-09-16): the tensor inventory that
-  decides a split, the one flag (`-ot`) that kills per-tensor placement across
-  devices, the two-device layer splits that DO work (pure-Vulkan and cross-vendor
-  CUDA+Vulkan, +7 % decode at 64k), and the harness-parsing trap that made the
-  first round of measurements say otherwise
-
-### Memory and Hardware
-
-- [[RAM CPU Offloading in VLLM]] — KV cache swap between GPU VRAM and system RAM
-- [[Hybrid CPU GPU Processing Dankserver]] — NVIDIA GPU compute with system RAM offload
-- [[Shared Memory Architecture Dankclaw]] — AMD unified memory and ROCm configuration
-- [[dankserver-vram-growth]] — VRAM/RAM growth behind the retired cuBLAS crash
-- [[Quantization by Hardware Architecture]] — which quant format fits which GPU/CPU
-
-### Retired Topology and History
-
-- [[Qwen3.8 PP Deployment]] — the retired pipeline-parallel head + worker
-- [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history (superseded by FastMTP)
-- [[FastFlowLM NPU Inference]] — the retired FastFlowLM NPU endpoint
-- [[DFlash DSpark Speculative Decoding]] — retired DFlash/DSpark drafting
-- [[GGUF Compatibility]] — which loaders accept which quant formats; the tensor-type histogram check
-- [[Llama Cpp Multi-Device Splits]] — layer vs tensor split across unlike devices; the -ot pipeline gate
-- [[Llama Cpp Serving Sizing]] — VRAM arithmetic from two measured boots, never a formula
-- [[PCIe Link and Atomics]] — link caps vs loaded rate; RCCL atomics routing on the old board
-- [[RDNA4 Container Serving]] — the serving stack recipe: mounts, ulimits, udev, verify gates
-- [[RDNA4 MoE Expert Offload VRAM Budget]] — arena sizing law for expert offload: free VRAM is estimator slack, not allocatable headroom
-- [[vLLM Boot Forensics and Config Interrogation]] — journal/endpoint forensics for boot failures; argv recovery from a live engine
-- [[vLLM Porting and Compile Triage]] — forward-porting vLLM majors; compile failure forensics
-- [[ling-3.0-flash-trial]] — the Ling-3.0-Flash trial runbook
-- [[ling-3.0-flash-vllm-plan]] — the Ling-3.0-Flash vLLM plan (retired)
-
-### vLLM Reference (historical)
-
-- [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
-- [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
-- [[vllm_research_findings]] — research findings from the vLLM era
-
-### Pipeline Parallelism (archive)
-
-- [[Ethernet Bandwidth Math]] — the bandwidth budget for the PP pair
-- [[Expected Performance]] — expected PP throughput
-- [[llama.cpp RPC]] — the RPC transport between head and worker
-- [[Mixed-Vendor Experiments]] — cross-vendor PP experiments
-- [[Qwen3.6 27B Spec Methods]] — spec-decoding methods for the 27B
-- [[Sample Script Set]] — reference scripts for the PP pair
-- [[vLLM Pipeline Parallelism]] — vLLM's PP implementation
-
-## Update Log
-
-- [[log]] — chronological record of knowledge-base changes
-
-## See Also
-
-- [AGENTS.md](../AGENTS.md) — repository rules
-- [serverAppConfigs-docs/LLM/index.md](LLM/index.md) — the LLM fleet knowledge base
-- [dankware Configuration Knowledgebase](~/.pi/agent/docs/index.md) — the "root" level system knowlegebase