dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(vault): one page per endpoint, forever -- 45 era pages fold into three

Browse Source 
Every payload swap a live endpoint has ever run is now one row in that
endpoint's Eras table, with its rollback path in the same row. No per-attempt
page, no per-swap ledger, nowhere else.

- Added: LLM/frank-archon.md (radiance, 2x R9700 TP=2), LLM/frank-dragoon.md
  (halogen flash-server, 19 eras), LLM/frank-zealot.md (llama.cpp PrismML fork
  on the 3090). Each carries Current shape, Eras, Laws, wire contract, files.
- Deleted: 45 per-payload runbooks -- the frank-dragoon line (11 including the
  pre-name halogen/kingjones/kairic/CIRU/eemin/agentionai/dflash2 pages and the
  benchmark page), the frank-zealot line (llama.cpp/vLLM/ik_llama/SGLang eras),
  the frank-archon line (vLLM/FreeToken/R9V/llama.cpp/radiance eras), plus
  frank-medium, frank-npu and the ubatch/PCIe one-offs that owned nothing.
- Rewired: every wikilink and markdown-path reference to a folded page points
  at the endpoint page instead; the three Research see-also lists and the
  ubatch sweep's runbook link included. No anchored links existed in the set.
- Regenerated both index pages as pure locators -- one row, one clause, no
  numbers -- so the docs_completeness locator law holds for all 153 pages.

tests/docs_completeness.bats green: index links every page, no unresolved
links. The pages moved to .AGENTTRASH/20261018-doc-dedupe/LLM/ (gitignored,
and git history holds every byte).
This commit is contained in:
Frank
2026-10-19 00:49:03 -07:00
parent af5ddfc54c
commit 219a097b39
88 changed files with 1211 additions and 11915 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/DFlash DSpark Speculative Decoding.md
+3 -3
View File
@@ -250,7 +250,7 @@ docker run --rm --name vllm-qwen36-dflash \
## Deployment on Dankclaw (AMD Strix Halo gfx1151)
`frank-dragoon` on Dankclaw ran the llama.cpp pipeline-parallel head (Qwen3.8-27B-Uncensored Q6_K) on port 7770, RPC-split with the Dankserver worker. Spec was the standalone MTP draft over the ngram-mod fallback, 1 slot at 262144 ctx, q8_0/f16 KV, head GPU side ROCm0. This page is a historical record of that era (mid-August 2026); for the current deployment see [[kingjones-strix-rocmfp4]].
`frank-dragoon` on Dankclaw ran the llama.cpp pipeline-parallel head (Qwen3.8-27B-Uncensored Q6_K) on port 7770, RPC-split with the Dankserver worker. Spec was the standalone MTP draft over the ngram-mod fallback, 1 slot at 262144 ctx, q8_0/f16 KV, head GPU side ROCm0. This page is a historical record of that era (mid-August 2026); for the current deployment see [[frank-dragoon]].
History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 2026-08-15 after a ROCm illegal-memory-access crash under two concurrent 16k prompts) and before that bundled MTP (`draft-mtp`, retired same day). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`), the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`), and the DSpark draft (`Qwen3.8-27B-DSpark-Q8_0.gguf`) remain in `/home/dank/models/` as reference only.
@@ -315,7 +315,7 @@ not switch without re-measuring".
The A/B vehicle for ANY single-GPU llama.cpp knob (batch/ubatch, KV type,
slot split, quant rung — not just spec flags) is the transient systemd
drop-in procedure in [[frank-dragoon-benchmark]] and [[frank-zealot-benchmark]].
drop-in procedure in [[frank-dragoon]] and [[frank-zealot]].
## Is it worth it? (arithmetic before deployment)
@@ -419,4 +419,4 @@ measurement that would settle it.
- [[GGUF Compatibility]] — header parsing and custom tensor-type lock-in
- [[Llama Cpp Serving Sizing]] — slot-pool and drafter KV sizing arithmetic
- [[Llama Cpp Multi-Device Splits]] — where the drafter's KV actually lands
- [[frank-dragoon-benchmark]] and [[frank-zealot-benchmark]] — the single-GPU env drop-in A/B vehicle
- [[frank-dragoon]] and [[frank-zealot]] — the single-GPU env drop-in A/B vehicle
serverAppConfigs-docs/LLM/Dankclaw Improvements.md
+6 -6
View File
@@ -18,8 +18,8 @@ marked "verify first". Re-verified 2026-08-15/16; corrections marked below.
**Live today (2026-10-18).** The era described below — llama.cpp on the 8060S
iGPU behind :7770 — is retired. Dankclaw now serves
[[frank-dragoon-flash-server-v2-native]] (halogen-flash-server 0.16.2, port
7770) and [[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML fork on
[[frank-dragoon]] (halogen-flash-server 0.16.2, port
7770) and [[frank-zealot]] (llama.cpp PrismML fork on
the RTX 3090, port 7771 — drift today: that unit is `failed` since 2026-10-18
03:40 and a hand-run 9B-MTP process answers the port), and boots `iommu=pt` since 2026-10-18
([[dankclaw-iommu-translation-off]]). The numbers below are the measurements of
@@ -32,7 +32,7 @@ re-checked 2026-10-18 and still hold.
(b10433): Vulkan 25.13 t/s vs ROCm 24.33 t/s decode at 16k; at 262k ctx
Vulkan decodes 48% faster (5.68 vs 3.83 t/s). ROCm wins prefill (410 vs
302 t/s @64k). The "HIP out-performs Vulkan for decode" claim did not
hold on the 8060S. See [[frank-dragoon-benchmark]].
hold on the 8060S. See [[frank-dragoon]].
- **Power profile**: the `/sys/class/drm/card0/device/power_profile` file
does not exist on this machine (checked 2026-08-15). GPU DPM is already
`performance`, forced level `auto`. This item is N/A.
@@ -67,7 +67,7 @@ The TTM row above is still live: `pages_limit = 29883536` (114 GiB) and
2026-10-18).
This split is the Strix Halo GTT/VGM design. It drives the batch/ubatch memory
limits documented in [[frank-dragoon-benchmark]].
limits documented in [[frank-dragoon]].
## 1. Self-Hosted Agentic Tools
@@ -130,7 +130,7 @@ All require root and a reboot for the modprobe change.
## Sources
- [[frank-dragoon-benchmark]] — measured backend/spec numbers
- [[frank-dragoon]] — measured backend/spec numbers
- [[deployment]] — deploy and update process
- AMD ROCm RDNA3.5 system optimization guide (gfx1151 memory/BIOS)
- llama.cpp source: `ggml/CMakeLists.txt`, `ggml/src/ggml-hip/CMakeLists.txt`,
@@ -139,7 +139,7 @@ All require root and a reboot for the modprobe change.
## See Also
- [[frank-dragoon-benchmark]]
- [[frank-dragoon]]
- [[DFlash DSpark Speculative Decoding]]
- [[deployment]]
- [[index]]
serverAppConfigs-docs/LLM/Dankclaw System Specs.md
+5 -5
View File
@@ -10,7 +10,7 @@ timestamp: 2026-09-03
## Server Overview
Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp inference through 2026-08-22 ([[ciru-activefpx-promptforge]], retired). The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same physical RAM pool. Today it serves [[frank-dragoon-flash-server-v2-native]] (halogen-flash-server 0.16.2 on the iGPU, port 7770) and [[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML fork on the RTX 3090, port 7771).
Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp inference through 2026-08-22 ([[frank-dragoon]], retired). The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same physical RAM pool. Today it serves [[frank-dragoon]] (halogen-flash-server 0.16.2 on the iGPU, port 7770) and [[frank-zealot]] (llama.cpp PrismML fork on the RTX 3090, port 7771).
## Hardware Specifications
### CPU
@@ -46,7 +46,7 @@ Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp
| Runtime | None installed. FastFlowLM 0.9.45 + XRT (installed 2026-08-16) was **retired 2026-09-11** with the `amd_iommu=off` decision — see [[dankclaw-iommu-translation-off]]; the package remains at `/home/dank/GitPackages/fastflowlm_0.9.45_ubuntu26.04_amd64.deb` |
| Status | Not in service — no NPU endpoint is deployed. The device itself is usable since the 2026-10-18 `iommu=pt` boot (`/dev/accel/accel0` present, `crw-rw---- root:render 261,0`); Gitea issue #3 tracks enablement. See [[FastFlowLM NPU Inference]] |
The NPU endpoint is retired (2026-09-11; [[frank-npu]]). The `frank-zealot` name now serves **Ternary-Bonsai-2-27B-PQ2_0** on the RTX 3090 via the llama.cpp PrismML fork (see [[frank-zealot-llama-prism-bonsai2-pq20]]); the Gemma4-12B QAT rung it once pointed at is retired history ([[frank-zealot-gemma4-12b]], 2026-09-17 → 10-01).
The NPU endpoint is retired (2026-09-11; [[frank-npu]]). The `frank-zealot` name now serves **Ternary-Bonsai-2-27B-PQ2_0** on the RTX 3090 via the llama.cpp PrismML fork (see [[frank-zealot]]); the Gemma4-12B QAT rung it once pointed at is retired history ([[frank-zealot]], 2026-09-17 → 10-01).
### ROCm
@@ -95,21 +95,21 @@ cable between claw and server is required for 10G.
| Kernel | 7.0.0-38-generic (`uname -r`, 2026-10-18) |
| Boot line | `iommu=pt` since 2026-10-18 (`/proc/cmdline`, no `amd_iommu` token); eras before: `amd_iommu=off` 2026-09-11 → 10-18, IOMMU default-on before that — [[dankclaw-iommu-translation-off]] |
| Python | 3.14.4 (system, `python3 --version`, 2026-10-18). The 3.12.13 CIRU venv left with that runtime (retired 2026-08-22) |
| CIRU runtime | ciru-ai/ROCmFPX — **retired 2026-08-22** ([[ciru-activefpx-promptforge]]); not the serving runtime |
| CIRU runtime | ciru-ai/ROCmFPX — **retired 2026-08-22** ([[frank-dragoon]]); not the serving runtime |
| uv | 0.12.8 (`~/.local/bin/uv --version`, 2026-10-18) |
## Runtime Relevance
- Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
- The GPU budget is the GTT aperture: ~114 GiB (`mem_info_gtt_total` in `/sys/class/drm/card1/device/` = 122,402,963,456 B; `card1` is the amdgpu iGPU, `PCI_ID=1002:1586`; `llama-server --list-devices` reports 116,732 MiB). rocm-smi's smaller SVM/dedicated carve-out is **not** the budget — the 2026-08 "64 GiB rocm-smi" lesson is stale. Do not use `free` or `dmidecode` for the GPU budget.
- The active services are `frank-dragoon` on 7770 — **halogen-flash-server 0.16.2** serving the native v2 abliterated `.hgn` in a rootless podman container ([[frank-dragoon-flash-server-v2-native]]) — and `frank-zealot` on 7771 ([[frank-zealot-llama-prism-bonsai2-pq20]]). The gufo v0.1.1 / UD-Q8_K_XL runtime this page once described is retired (2026-09-28 → 10-01, [[frank-dragoon-gufo-qwen38-27b]]); the gufo build tree is still on disk at `/home/dank/GitPackages/gufo-v0.1.1/`.
- The active services are `frank-dragoon` on 7770 — **halogen-flash-server 0.16.2** serving the native v2 abliterated `.hgn` in a rootless podman container ([[frank-dragoon]]) — and `frank-zealot` on 7771 ([[frank-zealot]]). The gufo v0.1.1 / UD-Q8_K_XL runtime this page once described is retired (2026-09-28 → 10-01, [[frank-dragoon]]); the gufo build tree is still on disk at `/home/dank/GitPackages/gufo-v0.1.1/`.
- The gufo-era note that the active runtime links against the host ROCm (`/opt/rocm` → `/etc/alternatives/rocm` → 7.2.4) through its RPATH belongs to that retired source build. The serving runtimes do not depend on the host ROCm that way: halogen runs inside its digest-pinned image, and `frank-zealot` is a CUDA build on the 3090. The retired CIRU venv's TheRock 7.15 SDK (`LD_LIBRARY_PATH`) was never in the serving path either.
- The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18.
- 424 GB free on `/` (`df -h`, 2026-10-18) is ample for model storage.
## See Also
- [[ciru-activefpx-promptforge]]
- [[frank-dragoon]]
- [[Shared Memory Architecture Dankclaw]]
- [[frank-redo-2026-08-17]]
- [[FastFlowLM NPU Inference]]
serverAppConfigs-docs/LLM/Dankserver System Specs.md
+4 -4
View File
@@ -16,7 +16,7 @@ timestamp: 2026-09-20
## Server Overview
Dankserver is a media/library server and the `frank-archon` LLM host (radiance
1.0.13, 2×R9700 tensor-parallel — [[frank-archon-radiance-qwen38-iq4r]]; the
1.0.13, 2×R9700 tensor-parallel — [[frank-archon]]; the
`frank-medium` name this page carried was retired 2026-09-20, see §Active
Services). The host runs Ubuntu 26.04 (Resolute Raccoon). The RTX 3090 that
served the FreeToken/CUDA `frank-zealot` era was removed on 2026-09-15 and
@@ -101,7 +101,7 @@ RCCL-backed rebuild needs no new host packages.
| Kernel | 7.0.0-38-generic, `iommu=pt` (`uname -r`, `/proc/cmdline`, read 2026-10-18) |
| Python | 3.14.4 |
| ROCm userspace | Ubuntu 7.1 packages in /usr (hipcc 7.1.1, rocminfo, rocm-smi 7.1.1) |
| Serving stack | radiance 1.0.13 container (`stilldeadcode/radiance@sha256:4c3f484564…`), TP=2 across both R9700s — [[frank-archon-radiance-qwen38-iq4r]]. The llama.cpp `fb27a525` build at `~/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714` is the retired frank-medium artifact |
| Serving stack | radiance 1.0.13 container (`stilldeadcode/radiance@sha256:4c3f484564…`), TP=2 across both R9700s — [[frank-archon]]. The llama.cpp `fb27a525` build at `~/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714` is the retired frank-medium artifact |
The stale AMD-repo ROCm 6.3.3 packages (167, `~24.04`-suffixed) that used to
hijack `libhsa-runtime64` were purged on 2026-09-15 — see
@@ -113,7 +113,7 @@ The active inference service is `frank-archon.service` (`systemctl is-active`
`active`, `is-enabled` → `enabled`, read 2026-10-18): radiance 1.0.13 in a
host-network podman container answering `curl 127.0.0.1:7770/health` →
`{"status":"ok","model":"frank-archon",…}`. See
[[frank-archon-radiance-qwen38-iq4r]].
[[frank-archon]].
The services this page once called active are gone: `frank-medium.service`
(llama-server, 32:33 layer split + DFlash2 drafter) is `not-found` — that rung
@@ -146,7 +146,7 @@ unverified. See [[deployment]].
## See Also
- [[frank-archon-radiance-qwen38-iq4r]] — the active serving runbook
- [[frank-archon]] — the active serving runbook
- [[Dual-R9700-Cooperation]] — why the PCIe path matters, and what the swap changed
- [[deployment]] — the deploy flow
- [[Shared Memory Architecture Dankclaw]] — for contrast, Dankclaw's unified memory
serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md
+2 -2
View File
@@ -16,7 +16,7 @@ timestamp: 2026-09-16
Dankserver's two R9700s run **one tensor-parallel engine across both cards** —
today that is `frank-archon` under radiance 1.0.13, TP=2
([[frank-archon-radiance-qwen38-iq4r]]). When this page was written the endpoint
([[frank-archon]]). When this page was written the endpoint
was `frank-medium`: first llama.cpp's layer split
([[frank-medium-llamacpp-dflash2]], 2026-09-16 → 09-20), then from 2026-09-20
one vLLM-radiance process at `tensor_parallel_size 2`
@@ -51,7 +51,7 @@ What that bought:
decode and 3.6x at 4-way concurrency, and it replaced the layer split on the day
of measurement. The payload has moved since: the MXFP4 `frank-medium` shape is
retired 2026-09-26, and what runs today is the iq4r publisher under radiance
1.0.13 ([[frank-archon-radiance-qwen38-iq4r]]).
1.0.13 ([[frank-archon]]).
- **The drafter, not the collective, is what llama.cpp lost.** Its tensor mode
could not carry the DFlash2 drafter, which is the honest caveat on the
comparison below.
serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md
+1 -1
View File
@@ -24,7 +24,7 @@ timestamp: 2026-09-03
> usable again since the 2026-10-18 `iommu=pt` boot (`amdxdna` bound,
> `/dev/accel/accel0` present — [[dankclaw-iommu-translation-off]]), but no
> service runs on it. `frank-zealot` today is the llama.cpp PrismML endpoint on
> the RTX 3090 ([[frank-zealot-llama-prism-bonsai2-pq20]]). See [[frank-npu]]
> the RTX 3090 ([[frank-zealot]]). See [[frank-npu]]
> for the retirement record. Everything below is the 2026-08 research record.
# FastFlowLM NPU Inference
serverAppConfigs-docs/LLM/GGUF Compatibility.md
+1 -1
View File
@@ -119,5 +119,5 @@ re-download.
- [[Llama Cpp Runtimes]] — which fork ships which custom types
- [[Llama Cpp Code Diff]] — the enum tables and code-level differences
- [[frank-dragoon-rocmfpx-q8agent-mtp]] — the ftype-115 lock-in in production
- [[frank-dragoon]] — the ftype-115 lock-in in production
- [[Llama Cpp Serving Sizing]] — pricing what a payload costs once it loads
serverAppConfigs-docs/LLM/Hybrid CPU GPU Processing Dankserver.md
+1 -1
View File
@@ -10,7 +10,7 @@ timestamp: 2026-08-28
## Overview
Dankserver has **no NVIDIA GPU**: the RTX 3090 left the box on 2026-09-15 and the host now runs two AMD Radeon AI PRO R9700s (`lspci | grep -ci nvidia` → 0, read 2026-10-18 — [[Dankserver System Specs]]). This page's vLLM-era hybrid guidance is reference material, not the serving path: the live service is radiance 1.0.13 as `frank-archon`, and its hybrid mechanism is a pinned host pool (43.88 GiB per rank, `scripts/frank-archon.env:184`) plus the CPU-resident 47.68 GiB n-gram/PLE table, not vLLM KV swapping ([[frank-archon-radiance-qwen38-iq4r]]).
Dankserver has **no NVIDIA GPU**: the RTX 3090 left the box on 2026-09-15 and the host now runs two AMD Radeon AI PRO R9700s (`lspci | grep -ci nvidia` → 0, read 2026-10-18 — [[Dankserver System Specs]]). This page's vLLM-era hybrid guidance is reference material, not the serving path: the live service is radiance 1.0.13 as `frank-archon`, and its hybrid mechanism is a pinned host pool (43.88 GiB per rank, `scripts/frank-archon.env:184`) plus the CPU-resident 47.68 GiB n-gram/PLE table, not vLLM KV swapping ([[frank-archon]]).
## Architecture
serverAppConfigs-docs/LLM/Llama Cpp Serving Sizing.md
+2 -2
View File
@@ -87,7 +87,7 @@ co-tenants.
(a retired frank-zealot shape: 22.6 of 24.1 GiB for an 11.29 GiB model; the card
today reads 24,576 MiB = 24.0 GiB total, `nvidia-smi --query-gpu=memory.total`,
2026-10-18, and the deployed payload is the PQ2_0 pair
[[frank-zealot-llama-prism-bonsai2-pq20]]). Do the capacity
[[frank-zealot]]). Do the capacity
subtraction before designing: 12 attn layers × 1.62 GiB + 200k f16 KV =
28.6 GiB > 24.1 GiB usable → KV-in-VRAM on that pair only reaches ~64k.
- **Context lengths: derive, never remember.** `llama-bench` computes
@@ -168,7 +168,7 @@ in the timings. Context cap for full-model residency on that shape stayed
## Related
- [[GGUF Compatibility]] — what a payload is before it is priced
- [[frank-zealot-llama-mimo-q6k-2x200704]] — the two-boot priced shape of the retired MiMo era (live shape today: [[frank-zealot-llama-prism-bonsai2-pq20]])
- [[frank-zealot]] — the two-boot priced shape of the retired MiMo era (live shape today: [[frank-zealot]])
- [[frank-medium-llamacpp-dflash2]] — where the 68 KiB/token datum was measured
- [[DFlash DSpark Speculative Decoding]] — the spec A/B protocol on llama.cpp
- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — the multi-device side
serverAppConfigs-docs/LLM/Model Swap and Rollback.md
+1 -1
View File
@@ -285,4 +285,4 @@ predecessor "stopped" (the port race above).
- [[Llama Cpp Serving Sizing]] — pricing the shape once it loads
- [[RDNA4 MoE Expert Offload VRAM Budget]] — vLLM-side fit arithmetic
- [[LLM Bench Windows and Measurement]] — measuring the new deployment
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the retired tcclaviger vLLM endpoint (2026-09-29 → 10-05); the live Dankserver endpoint is [[frank-archon-radiance-qwen38-iq4r]]
- [[frank-archon]] — the retired tcclaviger vLLM endpoint (2026-09-29 → 10-05); the live Dankserver endpoint is [[frank-archon]]
serverAppConfigs-docs/LLM/PCIe Link and Atomics.md
+1 -1
View File
@@ -253,4 +253,4 @@ kernel's declared args (`roc-obj-extract` + `llvm-readelf --notes`), look for
- [[Llama Cpp Multi-Device Splits]] — the layer-split path around the wall
- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — measured history
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the gather economics of the retired tcclaviger vLLM era (2026-09-29 → 10-05)
- [[frank-archon]] — the gather economics of the retired tcclaviger vLLM era (2026-09-29 → 10-05)
serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.8 PP Deployment.md
+2 -2
View File
@@ -17,7 +17,7 @@ timestamp: 2026-09-03
> removed from Dankserver on 2026-09-15 (it now sits in Dankclaw) and the server
> runs 2× R9700. The live multi-GPU scheme on Dankserver is tensor parallelism
> inside one process ([[Dual-R9700-Cooperation]],
> [[frank-archon-radiance-qwen38-iq4r]]).
> [[frank-archon]]).
# Qwen3.8-27B Pipeline-Parallel Deployment (Dankclaw + Dankserver)
@@ -199,7 +199,7 @@ exists** (checkout read 2026-10-18: `frank-pp-worker.env`,
`serve-pp-worker.sh`, `serve-pp-head.sh`, `frank-pp-worker.service`,
`pp-test.sh`, `pp-bench.py` and `prune-rpc-cache.sh` are all absent), and
`scripts/frank-dragoon.env` is today the halogen-flash-server env, not a PP
head env ([[frank-dragoon-flash-server-v2-native]]).
head env ([[frank-dragoon]]).
## Gotchas (all verified in b10433 source / live tests, 2026-08-15)
serverAppConfigs-docs/LLM/Pipeline Parallelism/index.md
+1 -1
View File
@@ -65,6 +65,6 @@ graph LR
## See Also
- [[DFlash DSpark Speculative Decoding]]
- [[frank-zealot-benchmark]]
- [[frank-zealot]]
- [[Dankserver System Specs]]
- [[Dankclaw System Specs]]
serverAppConfigs-docs/LLM/Quantization Production.md
+5 -5
View File
@@ -253,8 +253,8 @@ top-8**, moe_intermediate 512, hidden 2048, MTP ×1, 262k ctx, vocab
- Repo: `scripts/convert-swift15-w4dn8.sh`,
`scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`,
`scripts/kld-gate-swift15.sh`, `scripts/build-llama-rdna4.sh`,
`scripts/build-kairic-edge-gfx1151.sh`, [[agentionai-fast-imatrix]],
[[kingjones-strix-rocmfp4]], [[qwen3.8-strix-optimizations]],
[[frank-archon-radiance-swift15-w4dn8]],
[[frank-zealot-llama-qwen35-9b-q6k-2x200704]],
[[frank-dragoon-flash-server-v2-swift15]], [[GGUF Compatibility]].
`scripts/build-kairic-edge-gfx1151.sh`, [[frank-dragoon]],
[[frank-dragoon]], [[qwen3.8-strix-optimizations]],
[[frank-archon]],
[[frank-zealot]],
[[frank-dragoon]], [[GGUF Compatibility]].
serverAppConfigs-docs/LLM/RDNA4 Container Serving.md
+5 -5
View File
@@ -13,7 +13,7 @@ Bring-up path for Dankserver (2x Radeon AI PRO R9700, gfx1201, 31.86 GiB each =
`docker.io/tcclaviger/vllm` (vLLM 0.29.0.dev, GLM/Qwen-specialised RDNA4 build
with native MoE quant kernels and a first-class expert offload planner). The
tuning history and current deployed shape live in
[[dankserver-qwen38-fp8-expert-offload-tuning]]; this page is the bring-up law.
[[frank-archon]]; this page is the bring-up law.
## 0. Know the box — and its phantom third GPU
@@ -172,7 +172,7 @@ the REMAINDER to the expert arena. Consequences, all measured:
- **`--expert-offload-mem` caps the HOST side, not the VRAM arena**
(`cap = int(x * GIB)` against `host_memory_available()`). Do not reach for
it to fix a VRAM problem; to grow the arena, shrink the KV pin or raise
util (see [[dankserver-qwen38-fp8-expert-offload-tuning]] for the
util (see [[frank-archon]] for the
util=1.0 unbootability law).
**Read the plan — it is the real acceptance test:**
@@ -238,13 +238,13 @@ sequential STEPS rather than making each cheaper.
boolean (default 0; at depth 1 only ~10 % cold-vs-warm difference — the cache
already works); `CLAV_TUNABLEOP_SWEEP=1` autotunes GEMM shapes (silently
skipped under expert offload unless `CLAV_TUNABLEOP_WITH_OFFLOAD=1` too —
see [[dankserver-qwen38-fp8-expert-offload-tuning]] for why it was rejected);
see [[frank-archon]] for why it was rejected);
`--expert-precise-memory` swaps the fitted overhead figure in.
## Related
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — deployed shape, plan law,
- [[frank-archon]] — deployed shape, plan law,
measured campaigns
- [[PCIe Link and Atomics]] — why decode is gather-bound and what cannot fix it
- [[RDNA4 MoE Expert Offload VRAM Budget]] — arena/plan arithmetic deep-dive
- [[frank-dragoon-benchmark]] and [[frank-zealot-benchmark]] — window discipline on live hosts
- [[frank-dragoon]] and [[frank-zealot]] — window discipline on live hosts
serverAppConfigs-docs/LLM/RDNA4 MoE Expert Offload VRAM Budget.md
+2 -2
View File
@@ -11,7 +11,7 @@ timestamp: 2026-10-05T00:00:00-07:00
For VRAM-bound vLLM on the tcclaviger RDNA4 fork (frank-archon: Qwen3.8-
Flash-Next FP8, TP=2, 2x R9700 at 31.86 GiB each, expert offload to host RAM).
Tuning history and the deployed shape live in
[[dankserver-qwen38-fp8-expert-offload-tuning]]; boot-failure capture lives in
[[frank-archon]]; boot-failure capture lives in
[[vLLM Boot Forensics and Config Interrogation]].
## 1. The plan law
@@ -215,5 +215,5 @@ at graph capture; 6 seqs fp8 → 10.70 GiB ask with 9.18 GiB free (fails) —
- [[RDNA4 Container Serving]] — the bring-up path and container traps
- [[vLLM Boot Forensics and Config Interrogation]] — capture and contracts
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — measured history
- [[frank-archon]] — measured history
- [[Llama Cpp Serving Sizing]] — the llama.cpp equivalent
serverAppConfigs-docs/LLM/Shared Memory Architecture Dankclaw.md
+1 -1
View File
@@ -233,7 +233,7 @@ FP8 quantization depends on GPU architecture:
For FP4 on Dankclaw the ROCmFPX llama.cpp fork (`Q2_0_ROCMFP4`) provided native
ROCm FP4 kernels; that runtime line is **retired** (CIRU/Kairic eras,
[[ciru-activefpx-promptforge]], [[frank-dragoon-kairic-edge-qwen38-27b]]) and no
[[frank-dragoon]], [[frank-dragoon]]) and no
ROCmFPX build serves this host today.
## Example Commands
serverAppConfigs-docs/LLM/VLLM Index.md
+1 -1
View File
@@ -9,7 +9,7 @@ timestamp: 2026-08-28
# vLLM Index
> **RETIRED (2026-08-18).** The vLLM service on Dankclaw was replaced by the CIRU ROCmFPX llama.cpp runtime (v3). These pages are historical reference. See [[ciru-activefpx-promptforge]].
> **RETIRED (2026-08-18).** The vLLM service on Dankclaw was replaced by the CIRU ROCmFPX llama.cpp runtime (v3). These pages are historical reference. See [[frank-dragoon]].
## Documentation Overview
serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md
-217
View File
@@ -1,217 +0,0 @@
---
type: reference
title: agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw
description: agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF served by the charlie12345/ROCmFPX llama.cpp runtime (build-qwen38). 87.06 GiB Q4_0_ROCMFP4_FAST imatrix (v2 joined PLE layout), own MTP draft head (ROCmFP4-FAST, 2.28 GiB). frank-dragoon.
tags:
- llama.cpp
- rocm
- dankclaw
- gfx1151
- agentionai
- rocmfp4
- strix-halo
- qwen4exp
- mtp
timestamp: 2026-08-29
---
# agentionai Qwen3.8-Flash-Next ROCmFP4-FAST imatrix on Dankclaw
> **RETIRED 2026-08-30** — replaced by the kingjones STRIX model on the
> card's pinned kingjones30/ROCmFPX runtime; see
> [[kingjones-strix-rocmfp4]] for the active runbook. This page stays as
> the deployment record: the v2-joined-PLE layout requirement, the MTP
> head incompatibility (verified 3x), and the QSA F32-mask patch
> (`scripts/patches/qwen4exp-qsa-f32-mask.patch`) that made
> `--flash-attn on` work on the charlie build-qwen38 tree — all still
> true for that binary + model pair, which remains the rollback target.
`frank-dragoon` (port 7770) served
`agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF` (main,
`Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf`, the `v2/` joined layout)
with the author's own MTP draft head
(`Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf`, never loaded) on the
charlie12345/ROCmFPX `build-qwen38` runtime, 2026-08-29 → 2026-08-30.
## Why this model
- 87.06 GiB at 4.23 bpw — everything on the GPU of the 128 GB unified-memory
box, including the 51.2 B-parameter n-gram (PLE) table. REPO `v2/` layout
is used (joined single `per_layer_token_embd.weight` tensor) because the
charlie12345/ROCmFPX loader requires it; the root `-ple16` file's per-head
split only loads in the LaurentZuijdwijk Vulkan fork (see Known issues #2).
- imatrix-calibrated on 1540 chunks; +2.48% PPL vs reference on wikitext-2
(better than AesSedai IQ4_XS / IQ3_S at smaller size).
- 18-25% faster prompt processing than a mainline quant of similar size
(356 vs 285 t/s at 2048 tok; 301 vs 256 at 8k depth).
- Ships the model's own f16 vision tower (`mmproj/`).
- The MTP head is quantized to match the target (ROCmFP4-FAST) — a Q8_0
draft measured worse on this head family — but it does not load on
build-qwen38 (Known issues #3).
## Runtime
charlie12345/ROCmFPX `llama-server` built at
`/home/dank/GitPackages/ROCmFPX/build-qwen38` (HIP, ROCm 7.2.4, gfx1151,
PR #27742 qwen4exp merged). Upstream llama.cpp cannot load the ROCmFP4
quant types (`TYPE_100`/`TYPE_101`/FAST variants) — verified:
```
strings build-qwen38/bin/llama-server | grep Q4_0_ROCMFP4_FAST
```
CIRU binaries (ciru-ai v1.1) do **not** carry the FAST types; do not point
this model at the CIRU runtime. The build previously served the kingjones
STRIX / STRIX_LEAN models; the agentionai-FAST pair is the first use of the
author's own matched MTP head on this binary.
## Files / download
| Item | Path on Dankclaw | Size |
|---|---|---|
| Main (v2 joined) | `/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/v2/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf` | 93,484,242,816 B (87.06 GiB) |
| Main (root ple16, unused) | `/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2-ple16.gguf` | 93,484,237,760 B (87.06 GiB) |
| Vision | `…/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf` | 904,004,128 B |
| Draft head | `/home/dank/models/mtp-Qwen3.8-Flash-Next-FAST/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST.gguf` | 2,444,519,296 B |
Draft head sha256 (verified against HF LFS):
`5046d69571bb35c699e19c7a00b36633c2fdfdd06d70482551744ec523a6b590`
Downloaded with a resumable plain `curl -L -C -` (Hugging Face `resolve/main`
URLs; no auth token needed for public repos). The mmproj lives in an `mmproj/`
subdirectory of the repo.
## Serving profile
Card sampling table (instruct, non-thinking) mixed with the validated
build-qwen38 profile:
```
llama-server \
--model Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-v2.gguf \
--mmproj mmproj-Qwen3.8-Flash-Next-f16.gguf \
--host 0.0.0.0 --port 7770 --jinja \
--device ROCm0 --gpu-layers 999 \
--split-mode none --fit off --flash-attn on \
--ctx-size 32768 --parallel 1 \
--batch-size 4096 --ubatch-size 4096 \
--cache-type-k f16 --cache-type-v f16 \
--threads 16 --threads-batch 16 \
--ctx-checkpoints 16 --cache-ram 8192 \
--cache-prompt --cache-idle-slots --metrics \
--temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 1.5 \
--reasoning on --reasoning-format deepseek --reasoning-budget -1 \
--chat-template-file chat-template.jinja \
--chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}' \
--spec-type ngram-mod
```
Key choices and deviations from the CIRU deployment:
- **`--fit off`** — auto-fit segfaults on qwen4exp hyper-connection config
(same as every Flash-Next deployment).
- **`--checkpoint-min-step` removed** — build-qwen38 does not implement it
(verified via `--help`); only `--ctx-checkpoints` is passed.
- **No PLE sidecar args** — the n-gram table is in-GGUF (joined
`per_layer_token_embd.weight` from the `v2/` layout), not a `ple/`
CIRUPLE1 sidecar. `--ple-sidecar`/`--ple-cache-mib` and the
`GGML_QWEN4EXP_PLE_*` / `GGML_CUDA_Q41_MOE_FORCE_J` env knobs are CIRU-
runtime-only and are gone.
- **`--mmproj` active** — vision tower included (f16, lossless).
- **`--flash-attn on` (restored 2026-08-29).** fa on previously aborted graph
build: the QSA indexer path (layers with attention compression) called
`ggml_fill` on the F16-cast kq mask (`GGML_ASSERT(a->type == GGML_TYPE_F32)`,
ggml.c:5371, backtrace through `build_attn_qsa`). Fixed in the fork source
by the QSA F32-mask patch — vendored at
`scripts/patches/qwen4exp-qsa-f32-mask.patch` (sha256
`f2f4a486e21cfad80bfa42683281d18e1cba22fa43c5e0e2aa0cfdbe141e925f`);
the binary at `build-qwen38/bin/llama-server` (rebuilt 2026-08-29) already
carries it. A clean rebuild must re-apply
`git apply scripts/patches/qwen4exp-qsa-f32-mask.patch` inside
`/home/dank/GitPackages/ROCmFPX` before `make`. Verified live at
CTX_SIZE=32768: model loads, health 200, real completion.
- Context capped at **32768 × 1 slot — the author's own quick-start**. This
model is fully GPU-resident (87.06 GiB): the PLE n-gram table sits inside
the GGUF as one tensor, unlike STRIX_LEAN which streamed it from RAM.
HIP heap is 119.3 GiB; the context buffer is 50.8 GiB at 131072 and
103 GiB at 262144, so either window OOMs (`cudaMalloc … failed: out of
memory`, verified live on first graph alloc). At 32768 the buffer is
~13 GiB (~100 GiB total) and loads clean.
- MTP is disabled (ngram-mod only) — see the MTP section below.
## MTP draft head — INCOMPATIBLE on build-qwen38
The author's own matched head (`agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-
FAST-GGUF`, 2.28 GiB, `qwen4exp` arch) does **not** load on this binary:
```
error loading model: missing tensor 'blk.0.hc_attn_norm.weight'
```
The build-qwen38 qwen4exp loader requires a full per-layer
`blk.<il>.hc_attn_norm.weight` set in the draft (the graph reads all 48
layers); the head only carries `blk.48.hc_attn_norm.weight` (count 1 — all
three Flash-Next-era heads examined on this fork have that same partial
layout). Verified three times, not a cross-vendor artifact:
1. EasiiX Q8_0 head vs kingjones STRIX_LEAN main (2026-08-28) — failed.
2. agentionai FAST head vs kingjones STRIX_LEAN main (2026-08-28) — failed.
3. agentionai FAST head vs agentionai v2 main (2026-08-29, this deployment)
— failed identically.
The 2.28 GiB head stays on disk; the moment a binary or head appears that
carries the full per-layer norm set, re-enable with
`SPEC_TYPE=draft-mtp,ngram-mod`, `--spec-draft-n-max 3` (author-measured
best: 32.4 t/s, 0.612 acceptance; adaptive p-split measured worse).
## Config files
- Env: `scripts/frank-dragoon.env`
- Serve: `scripts/serve-frank-dragoon.sh`
- Chat template: `scripts/chat-template.jinja` (unchanged, Qwen-Sharp)
- Fork patch: `scripts/patches/qwen4exp-qsa-f32-mask.patch` (re-apply after
any clean fork rebuild — see Serving profile)
## Swap procedure
```bash
sudo systemctl stop frank-dragoon
# edit scripts/frank-dragoon.env + scripts/serve-frank-dragoon.sh, commit, push
# on Dankclaw: git pull --ff-only && scripts/deploy.sh dankclaw --service frank-dragoon
sudo systemctl start frank-dragoon # or via deploy.sh
curl http://192.168.0.143:7770/health
```
Model load is slow (87 GiB + 2.3 GiB draft): health returns 503 while
loading; `deploy.sh` waits up to 20 minutes (`LLM_WAIT`). Do not kill it.
## Known issues
1. **ROCmFP4 types require the ROCmFPX fork.** Stock llama.cpp and the CIRU
runtime reject these files.
2. **Use the `v2/` joined layout, NOT the root `-ple16` split.** This binary's
qwen4exp loader (qwen4exp.cpp:104-108) unconditionally requires a single
`per_layer_token_embd.weight` tensor and GGML_ASSERTs "qwen4exp is missing
the PLE n-gram table" against the per-head split. The root file only loads
in the LaurentZuijdwijk Vulkan fork (`vulkan-qwen4exp-rocmfpx`). The v2
file is byte-identical in weights, ~5 KB larger header, loads fully into
SVM unified memory (no `--ngram-on-disk` needed — kingjones STRIX ran a
50 GiB joined PLE table the same way).
3. **No MTP draft head loads on build-qwen38.** The qwen4exp loader
requires a full per-layer `blk.<il>.hc_attn_norm.weight` set; every
Flash-Next-era head checked (EasiiX Q8_0, agentionai FAST, CIRU) carries
only `blk.48`. Spec decoding is ngram-mod only. See the MTP section.
4. **`--flash-attn on` (restored 2026-08-29).** fa on aborts graph build
without the QSA F32-mask patch — see Serving profile (fix + sha256
vendored there).
5. **`SVM mapping failed` kernel spam** is a documented symptom, not root
cause — investigate before blaming memory pressure.
## Citations
- [agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF)
- [agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF)
- [Qwen/Qwen3.8-Flash-Next base model](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)
- [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) — fork with PR #27742 (qwen4exp architecture) and the ROCmFP4 types
- [llama.cpp PR #27742](https://github.com/ggml-org/llama.cpp/pull/27742) — qwen4exp support (merged upstream)
- [llama.cpp PR #27739](https://github.com/ggml-org/llama.cpp/pull/27739) — MTP draft head support
serverAppConfigs-docs/LLM/ciru-activefpx-promptforge.md
-266
View File
@@ -1,266 +0,0 @@
---
type: reference
title: CIRU ActiveFPX PromptForge on Dankclaw
description: jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (v3) served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Install runbook, ROCm 7.15 toolchain, and the frank-dragoon service.
tags:
- llama.cpp
- rocm
- dankclaw
- gfx1151
- ciru
- promptforge
timestamp: 2026-08-22
---
# CIRU ActiveFPX PromptForge on Dankclaw
> **RETIRED 2026-08-24.** `frank-dragoon` served this v3 model from
> 2026-08-22 (restored after a same-day Kairic Edge IU4 experiment was
> reverted — operator decision, commit `c60156f`; see [[kairic-edge]])
> until [[eemin-rocmi4-mtp]] took over on 2026-08-24. Kept as the
> historical record of the v3 deployment.
`frank-dragoon` (port 7770, moved from 52625 on 2026-08-22) serves
`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` **v3** through the CIRU
ROCmFPX llama.cpp runtime. This replaced the retired vLLM service that
previously occupied the slot. The v2 deployment moved to v3 on 2026-08-21.
## What This Is
The model is a **GGUF**, not a vLLM checkpoint. v3 is the consolidated
quality-and-speed release of Qwen3.8-27B for AMD ROCm: an FFN-equalized
compact body, 50 retained late-writer Q6 tensors, full-Q8 output
projection, full-W8 PromptForge view (FFN + Gated DeltaNet), the
small-row W8 route, a draft-only Output-K8 shortlist proxy with Q8
reranking, native depth-4 MTP (no ngram), and the BF16 Qwen3.8 vision
tower + 5120-wide merger as a separate projector.
| Artifact | Size |
|----------|------|
| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v3-Q8-FFNEQ-LateQ6.gguf` | 15.44 GiB |
| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-FFN.pfs` | 15.95 GiB |
| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-GDN.pfs` | 3.75 GiB |
| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-Output-K8.pfs` | 0.67 GiB |
| `Qwen3.8-27B-mmproj-BF16.mmproj` | 0.87 GiB |
| Full v3 payload (GGUF + 3 .pfs + mmproj) | ~36.7 GiB |
The `.pfs` companions and the Output-K8 proxy are optional. Omit them to
run the GGUF alone and save ~20.6 GiB of memory. The GGUF remains
independently usable.
## Requirements
- **ROCm 7.15 is required** for the advertised Strix Halo performance.
The system ROCm (`/opt/rocm-7.2.4`) selects slower or fallback kernels.
- The pinned runtime is `ciru-ai/ROCmFPX` tag
`qwen3.8-activefpx-promptforge-v2.3` (commit `103af8b41`) **plus the
v3 runtime patch** `runtime/qwen38-v3-output-k8-runtime.patch`. On
Dankclaw the patch is commit `af0eac6` on top of the v2.3 checkout.
- Composable Kernel at `fdf4bb7fcc984811cef48ce817d89aac064b984a`.
- Validated GPU: AMD Radeon 8060S / gfx1151.
## ROCm 7.15 Toolchain
The retired `ling3-ciru` package left a complete TheRock ROCm 7.15 SDK
(`7.15.0a20260728`) inside its venv:
```
/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/
_rocm_sdk_devel/ # hipcc, amdclang 23, cmake, headers
_rocm_sdk_libraries/ # hipblas, hipblaslt, rocroller
_rocm_sdk_core/ # libamdhip64, libhsa-runtime64
rocm-sdk-device-gfx1151
```
The compiler is amdclang 23. `rocm-sdk version` reports `7.15.0a20260728`.
The binary is built against and run with these 7.15 libraries.
## Build Runbook (2026-08-18, v2.3; +v3 patch on 2026-08-21)
Step 1-4 below built v2.3 on 2026-08-18. On 2026-08-21 the v3
Output-K8 runtime patch was applied on top and the server rebuilt:
0. Apply the v3 runtime patch to the v2.3 checkout and rebuild:
```
% verify the patch hash: 8ba7d721ea380b060aeb0c7595b8b05f1582c0f7bf7a4b89e0910bc8e72465d5
git am runtime/qwen38-v3-output-k8-runtime.patch # -> commit af0eac6
cmake --build build-promptforge --target llama-server -j$(nproc)
```
The pre-built binary is kept as `build-promptforge/bin/llama-server.v2.3-bak`
for rollback.
1. Install host build tools (ninja-build was missing): `sudo apt install ninja-build`.
2. Clone the runtime and CK at their pinned revisions:
```
git clone --depth 1 --branch qwen3.8-activefpx-promptforge-v2.3 \
https://github.com/ciru-ai/ROCmFPX.git
git clone --depth 1 https://github.com/ROCm/composable_kernel.git
git -C composable_kernel checkout fdf4bb7fcc984811cef48ce817d89aac064b984a
```
3. CK headers are generated at CK configure time (`ck/config.h`,
`ck/version.h`). The llama.cpp build includes the CK source tree, so run
CK's configure once and copy the headers into `include/ck/`:
```
cmake -S composable_kernel -B build-ck -DAMDGPU_TARGETS=gfx1151 -DCK_ENABLE_WMMA=ON
cp build-ck/include/ck/config.h build-ck/include/ck/version.h composable_kernel/include/ck/
```
4. Configure the llama.cpp build against the venv ROCm 7.15 SDK:
```
export ROCM_PATH=$SDK/_rocm_sdk_devel
cmake -S ROCmFPX -B build-promptforge -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER=$ROCM_PATH/llvm/bin/clang \
-DCMAKE_CXX_COMPILER=$ROCM_PATH/llvm/bin/clang++ \
-DCMAKE_HIP_COMPILER=$ROCM_PATH/llvm/bin/clang++ \
-DCMAKE_PREFIX_PATH=$ROCM_PATH \
-DGGML_HIP=ON -DGGML_CUDA=OFF -DGGML_VULKAN=OFF \
-DGGML_HIP_FORCE_MMQ=ON -DGGML_HIP_GRAPHS=ON \
-DGGML_HIP_MMQ_MFMA=ON -DGGML_HIP_NO_VMM=ON \
-DGGML_HIP_ROCWMMA_FATTN=OFF -DGGML_NATIVE=ON \
-DAMDGPU_TARGETS=gfx1151 -DGPU_BUILD_TARGETS=gfx1151 \
-DPROMPTFORGE_CK_ROOT=$CK_ROOT \
-DLLAMA_BUILD_SERVER=ON \
-DLLAMA_BUILD_WEBUI=OFF
cmake --build build-promptforge --target llama-server -j$(nproc)
```
`LLAMA_BUILD_WEBUI=OFF` is required. The embedded browser UI downloads
assets from an HF bucket (`llama-ui`) that returns 404; the service is
consumed through the OpenAI-compatible API, so the UI is not needed.
Build output: `build-promptforge/bin/llama-server`.
```mermaid
graph TD
A[Clone ROCmFPX v2.3 + CK pinned] --> B[Configure CK, copy generated headers]
B --> C[Configure llama-server vs ROCm 7.15 SDK]
C --> D[Build llama-server]
D --> E[Verify --list-devices shows gfx1151]
E --> F[Download v3 GGUF + FFN/GDN/Output-K8 + mmproj]
F --> G[Deploy frank-dragoon]
G --> H[Health check + chat completion]
```
## Model Download
With the HF token to the v3 staging directory
`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/`. The v3 files live there
so the v2 artifacts in `/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX/` stay
intact for rollback. Download ran on Dankclaw with the background script
`/tmp/download-v3.sh`; all five files + the patch passed SHA-256 checks
against the model card before the service restart.
## Service
- Unit: `frank-dragoon.service`, port 7770
- Script: `scripts/serve-frank-dragoon.sh`, env `scripts/frank-dragoon.env`
- The script exports the ROCm 7.15 SDK libraries in `LD_LIBRARY_PATH` before
the system ROCm. `ROCM_PATH` points at the devel SDK.
### Launch profile
- 262144 context (native max), 1 slot (user request 2026-08-19; parallel
slots degraded performance), f16 KV (~16 GiB), context checkpoints (16),
prompt cache. Concurrent requests queue on the single slot; the server
never preempts a running generation
- Native MTP draft, depth 4, device ROCm0
- `--temp 0.7 --top-p 0.95`, reasoning on (`--reasoning-format deepseek`
puts `<think>` reasoning in `message.reasoning_content`)
- v3 runtime: Output-K8 draft proxy (top-8 shortlist, Q8 rerank) +
small-row W8 prompt route, full-W8 FFN/GDN views
- PromptForge env: `PROMPTFORGE_MODE=m2048_fused_tail1476`,
`GGML_CUDA_GRAPH_OPT=0`, `PROMPTFORGE_MTP_OUTPUT_K8=1`,
`PROMPTFORGE_MTP_OUTPUT_K8_VALIDATE=0`,
`PROMPTFORGE_ENABLE_SMALLM_W8=1`
### Required environment
```
export PROMPTFORGE_SIDECAR=...FFN.pfs
export PROMPTFORGE_GDN_SIDECAR=...GDN.pfs
export PROMPTFORGE_MODE=m2048_fused_tail1476
export GGML_CUDA_GRAPH_OPT=0
export PROMPTFORGE_ENABLE_SMALLM_W8=1
export PROMPTFORGE_MTP_OUTPUT_K8=1
export PROMPTFORGE_MTP_OUTPUT_K8_VALIDATE=0
export PROMPTFORGE_MTP_OUTPUT_K8_PROXY=...Output-K8.pfs
# v3 qualified profile: disable the CPU argmax draft fast path and any
# IU4/ngram lane. The Output-K8 proxy + backend sampling replace them.
unset LLAMA_MTP_CPU_ARGMAX_FASTPATH LLAMA_MTP_BACKEND_GREEDY
unset PROMPTFORGE_IU4_SIDECAR PROMPTFORGE_GDN_IU4_SIDECAR
unset PROMPTFORGE_ENABLE_NGRAM_MOD PROMPTFORGE_ENABLE_NGRAM_M65_IU4
```
The v3 qualified profile runs the MTP draft through the Output-K8 proxy,
not the CPU argmax fast path. The old `LLAMA_MTP_CPU_ARGMAX_FASTPATH=1`
profile is retired for this slot. A TOP_K sampler warning may still appear
in the journal once per start; the Output-K8 draft path does not depend on
it and generation proceeds.
## Operational Notes
- Health: `curl http://127.0.0.1:7770/health`. Returns 503 while loading.
- Model load is slow (v3 five-file payload ~36.7 GiB); do not kill the
process during warmup. The page cache makes a repeat load fast right after
download, but a later cold start takes several minutes.
- `deploy.sh dankclaw` waits 20 minutes for health.
- The systemd unit must not set `HOST` or `TEMP` env vars (HIP collision
rule). `--host`/`--port`/`--temp` are CLI flags only.
- v3 adds the BF16 vision tower over v2. The service reports `multimodal`
capability; mmproj loads with the language model (CLIP on ROCm0 backend).
## ngram-mod + MTP Stacking (benchmarked 2026-08-21)
This page records a research question: does ngram-mod add value when
stacked under the native MTP draft? The v3 profile runs MTP alone; the
benchmark below compares that against the stacked pair.
### Feasibility
ngram-mod DOES stack with MTP in this CIRU fork. The v3 build compiles in
both `ngram-mod` and `draft-mtp` (`common/speculative.cpp`). Pass
`--spec-type draft-mtp,ngram-mod` (or `ngram-mod,draft-mtp`) and the fork
initializes both implementations in the same process. The boot log shows:
```
adding speculative implementation 'ngram-mod'
initialized ngram_mod with n_match=24, size=4194304 (16.000 MB)
adding speculative implementation 'draft-mtp'
```
The model card's "no ngram subsystem" line refers to the CIRU PromptForge
env lanes (`PROMPTFORGE_*_NGRAM_*`, `PROMPTFORGE_ENABLE_NGRAM_MOD`), not
the upstream llama.cpp spec chain. Those env lanes must stay unset; the
`--spec-type ngram-mod` chain is separate and works.
### Ordering — the CLI order does not matter
The `--spec-type` string order is discarded. `common_speculative_init`
collapses the requested types into a bitmask, then rebuilds the chain in a
hard-coded priority: ngram types first, then draft-simple, then MTP.
So `draft-mtp,ngram-mod` and `ngram-mod,draft-mtp` produce the same
internal order (`ngram-mod` then `draft-mtp`) and the same runtimes.
Verified: both orders decode ~25.3 t/s at 8k.
### Results (decode generation t/s, 200 tokens, prose prompt)
| ctx | MTP only | MTP + ngram-mod | verdict |
|-----|----------|-----------------|---------|
| 8k | 29.42 | 25.27 | ngram hurts -4.15 |
| 16k | 26.42 | 21.13 | ngram hurts -5.29 |
| 64k | 19.72 | 18.00 | ngram hurts -1.72 |
At every depth, adding ngram-mod under MTP REDUCED decode throughput. The
ngram draft costs its own forward pass / occupancy without adding enough
accepted tokens on top of MTP. Baseline (no ngram) remains the production
config (`--spec-type draft-mtp` only). Keep ngram-mod off.
The historical harness was moved to `.AGENTTRASH/bench-scripts-20260901/` and is retained for audit only.
it boots the v3 stack on a scratch port and reads authoritative
`print_timing` lines. It requires the production service to be stopped.
## See Also
- [[frank-redo-2026-08-17]] — the fleet topologies this slot has served.
- [[Shared Memory Architecture Dankclaw]] — unified memory sizing.
- [[qwen3.8-strix-optimizations]] — llama.cpp ROCm0 baselines on this iGPU.
- [[VLLM Overview]] — vLLM reference (retired for this slot).
serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md
+3 -3
View File
@@ -205,7 +205,7 @@ answered 200. (Later the same day the pool DOUBLING to 401,408 positions for
slot parity was COMMITTED (`2b12716`) and **deployed the same evening**: the
boot now reserves `401408 positions (about 11.0 GiB)` and the engine-side total
is `79.6 GiB in all` (62.9 weights + 11.0 KV + 5.7 working),
see [[frank-dragoon-flash-server-v2-native]].)
see [[frank-dragoon]].)
**The NPU came back with the flip.** `amdxdna` bound to `0000:c6:00.1` (Strix/
Krackan/Strix Halo NPU rev 11), firmware `amdnpu/17f0_11/npu_7.sbin`, `Initialized
@@ -229,8 +229,8 @@ lives in `~/.local/bin`.
## Related
- [[frank-npu]] — the endpoint this retired, including the err=-19 root cause
- [[halogen-flash-server]] — the runtime bought by the prefill figure
- [[kingjones-strix-rocmfp4]] — the runtime it replaced (rollback target)
- [[frank-dragoon]] — the runtime bought by the prefill figure
- [[frank-dragoon]] — the runtime it replaced (rollback target)
## Citations
serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md
+3 -3
View File
@@ -166,7 +166,7 @@ Not advisable.
| Runtime | gfx1201 | Architectures | Note |
|---|---|---|---|
| **Affinity** | yes, built here | **V4-Flash only** — "the geometry is compiled in; there is no architecture abstraction" | 110 tok/s for 284B. Best engine for this box by an order of magnitude, wrong model |
| **tcclaviger vLLM fork** | **yes — proven 2026-09-29** | qwen4exp measured 37.4-40.6 tok/s decode (see [[dankserver-qwen38-fp8-expert-offload-tuning]]); **`glm5next` registered** (`Glm5NextForCausalLM`/`MTP`, backend-split `amd/`/`common/`/`nvidia/`, `sparse_indexer.py` for DSA); r4d.so has **native mxfp4 + dsfp4 MoE entries** (`has_dsfp4() == True`); W4A16 MoE class (`R4dW4a16MoEExperts`) wired into the same tiered offload; TunableOp silently skipped under offload (needs `CLAV_TUNABLEOP_WITH_OFFLOAD=1`, and then corrupts numerics — do not use) | the ad hoc qwen-fp8 container's runtime; **the old "MXFP4 = emulation/missing kernels" verdict predates this fork and is dead** — the GLM-5.3-Flash gate is now the payload format + a boot probe, not kernels |
| **tcclaviger vLLM fork** | **yes — proven 2026-09-29** | qwen4exp measured 37.4-40.6 tok/s decode (see [[frank-archon]]); **`glm5next` registered** (`Glm5NextForCausalLM`/`MTP`, backend-split `amd/`/`common/`/`nvidia/`, `sparse_indexer.py` for DSA); r4d.so has **native mxfp4 + dsfp4 MoE entries** (`has_dsfp4() == True`); W4A16 MoE class (`R4dW4a16MoEExperts`) wired into the same tiered offload; TunableOp silently skipped under offload (needs `CLAV_TUNABLEOP_WITH_OFFLOAD=1`, and then corrupts numerics — do not use) | the ad hoc qwen-fp8 container's runtime; **the old "MXFP4 = emulation/missing kernels" verdict predates this fork and is dead** — the GLM-5.3-Flash gate is now the payload format + a boot probe, not kernels |
| **Lucebox** | yes, image on host | Qwen 3.8, DS4; **V4.1 on the unmerged `feat/ds41-engine` branch** | The only V4.1 path. **Refuses layer split and layer-major prefill for V4.1** |
| **radiance-vLLM** (drwolfen) | yes — served frank-archon 2026-09-27 → 09-30 | welded to `qwen4_exp`; PLE / QSA / packed-HC live inside that model class | 12–20 person-days per architecture |
| **FreeToken-ROCm** | partial | block-FP8 safetensors only (modelopt); known `CpuMoeExecutor` issue under ROCm | cannot load a GGUF |
@@ -221,7 +221,7 @@ gfx1201+gfx1201. hip:0 takes dense weights and the hot expert stack;
## See Also
- [[frank-archon-lucebox-ds4]] — the retired Lucebox frank-archon runbook
- [[frank-archon]] — the retired Lucebox frank-archon runbook
- [[Dual-R9700-Cooperation]] — the PCIe/atomics facts behind the board swap
- [[Dankserver System Specs]] — the host (RAM corrected 2026-09-28)
- [[frank-archon-r9v-radiance-qwen38-q8]] — the ACTIVE frank-archon
- [[frank-archon]] — the ACTIVE frank-archon
serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md
-724
View File
@@ -1,724 +0,0 @@
---
type: research
title: Qwen3.8-Flash-Next FP8 on tcclaviger — expert-offload decode tuning
description: VERIFIED 2026-09-29 by measurement. The flag-level decode-tuning campaign on the ad hoc qwen-fp8 container (orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8, tcclaviger RDNA4 vLLM fork, 2x R9700): the >35 tok/s target is MET at 37.4-40.6 tok/s by turning prefetch OFF — telemetry measured its precision at 36%, wasting ~half of all gather traffic — while the original 26.8 tok/s incumbent (prefetch on) survived every other challenger. Residual upside is the fork's gather read path (effective ~6.5-9.5 GB/s vs 22.7 GB/s the link delivers at the same slab size).
tags:
- dankserver
- r9700
- gfx1201
- qwen3.8
- fp8
- expert-offload
- speculative-decoding
- tuning
timestamp: 2026-09-29
---
# Qwen3.8-Flash-Next FP8 — expert-offload decode tuning
> **Question this page answers.** On the tcclaviger RDNA4 vLLM fork serving
> `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` across both R9700s, what
> runtime flags actually move decode at a 32k working point?
>
> **Answer in one line.** **37.44–40.57 tok/s — the >35 target is met — by
> deleting `--expert-prefetch 1`.** Expert-cache telemetry measured the
> prefetcher at **36% precision** (36,720 slabs prefetched, 13,369 served):
> ~half of all gather traffic was useless copies, and the copies it "saved"
> would have been installs anyway. The prefetch-on incumbent (26.8 tok/s)
> survived every other challenger measured: gather geometry, spec depth,
> TunableOp, scheduler batch knobs, concurrency, ngram-only drafting.
This is the ad hoc `qwen-fp8` container on Dankserver (rootless podman,
image `docker.io/tcclaviger/vllm:latest`, engine
`v0.29.0.dev0+g2bdbbc8080`), NOT the `frank-archon` systemd service —
the R9V production unit is untouched, so no stop-and-mask was required for
any of this. All numbers are decode tok/s measured from the server's own
`usage.completion_tokens / (wall - ttft)` — SSE chunk counting is invalid
under speculative decoding because rejected drafts change the accounting.
## Deployment shape (the tuned incumbent)
Serve script on the host (`/home/dank/serve_qwen_fp8.sh`), env hooks in
CAPS. **PREFETCH must be unset** (omitting the flag; the fork's default is
off):
```
MODEL=/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8 (~172.8 GiB, 131 shards)
IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
# image ID 3322dbbbc96d; local alias qwen-fp8-runtime:tuned-20260929;
# the floating :latest tag moved once already — always launch by digest.
# repo tag of this state: qwen-fp8-tuned-20260929
--tensor-parallel-size 2 --max-model-len 32768 --gpu-memory-utilization 0.98
--kv-cache-memory-bytes 2147483648 --max-num-seqs 4 --enable-prefix-caching
--speculative-config '{"method":"mtp","num_speculative_tokens":4}'
# NO --expert-prefetch: measured 36% precision, ~50% wasted gather traffic
```
Published to the host LAN via the serve script's `PORTMAP` hook
(`PORTMAP=7773:7770`): health and completions answer on
`http://192.168.0.235:7773` (verified host-side 2026-09-29; 256k config,
decode 36.49/34.55/35.98 through the published path). Host port 7773 was
chosen over 7770 because `frank-archon` (7770) is stopped but ENABLED —
any future start would collide with a published 7770.
Device notes carried from the bring-up: `--device /dev/dri` must be the
WHOLE directory (individual renderD nodes enumerate but refuse contexts —
"No CUDA GPUs are available" while device_count() reports 2); the image
entrypoint execs `vllm serve` itself.
The memory plan this produces (from the engine's own `expert_plan.py` log):
| region | size |
|---|---:|
| expert cache space | 15.05 GiB = 6,574 slots x 2.344 MiB |
| &nbsp;&nbsp;VRAM-only resident layers | 3.52 GiB (layers 0-2, 1,536 experts) |
| &nbsp;&nbsp;swap region | 11.53 GiB (5,038 slots, 111-112/layer) |
| host-backed experts | 52.74 GiB per rank, 23,040 experts |
| KV reserve (one max-len request +8%) | 0.36 GiB of the 2 GiB pool |
| CUDA graphs | 1.93 GiB (21 graphs; 1.32 GiB at `--max-num-seqs 2`) |
| runtime overhead | 7.6 GiB (non-torch + buffers + activation) |
## Measured arms (2026-09-29)
Decode tok/s at 166 / 1087 / 4064 prompt tokens, median of 3 runs after a
64-token warm pass, temperature 0, max_tokens 256:
| Arm | 166 | 1087 | 4064 | Acceptance | Verdict |
|---|---:|---:|---:|---:|---|
| **MTP4, stock 8x16, PREFETCH OFF** | **40.57** | **37.44** | **37.57** | 54.0% | **NEW INCUMBENT — target met** |
| MTP4, stock 8x16, prefetch ON | 26.80 | 25.15 | 26.81 | 60.7% | old incumbent (+51%/+40% worse) |
| MTP4, prefetch ON (re-check) | 25.80 | 24.38 | 26.63 | 47.9% | reproduces (noise band ±1) |
| MTP4, prefetch ON, gather 16x32 | 24.30 | 22.77 | 24.11 | 50.3% | −9% — ruled out |
| MTP4, prefetch ON, gather 32x64 | 26.58 | 24.94 | 26.34 | 49.6% | parity — ruled out |
| MTP3, prefetch ON | 24.47 | 24.41 | 24.44 | 66.7% | −9%, flat in ctx — ruled out |
| MTP6, prefetch ON | 19.07 | 17.81 | 13.54 | 17.4% | collapses; crosses UVA threshold — ruled out |
| MTP4, prefetch ON, MNBT 8192 | 29.32 | 24.12 | 23.65 | — | neutral — ruled out |
| MTP4, prefetch ON, PREFETCH depth 2 | 25.83 | 25.06 | 25.87 | 56.0% | neutral — ruled out |
| MTP4, prefetch ON, TunableOp w/ offload | 23.59 | (79.82*) | (79.89*) | (96.5%*) | **rejected — corrupts** |
| 2-way concurrency, prefetch ON (2 streams) | — | — | 15.06 + 23.11 = **25.49 summed** | — | no gain vs 26.8 single |
| ngram-only drafter | — | — | — | — | fork bug: PLE inputs unprepared |
| 256k ctx (MAXLEN=262144, KV 4.5 / 4.03 GiB, prefetch off) | 36.65 / 36.48 | 34.21 / 34.13 | 34.77 / **35.02** | 53.0 / 39.4% | boots and serves; band 34.1-36.5 straddles 35 within noise; arena identical (5,648) at both KV sizes |
\* The TunableOp numbers are invalid by construction: the tuned GEMMs
changed generated behavior (repetition-loop, the server's
"repetition detected" halt firing at ~140-146 tokens instead of the full
256, acceptance jumping 60.7% → 96.5%), the 79.8 tok/s figures were
measured on those halted degenerate completions, and the text-only check on
a fresh prompt came back 20.01 tok/s — *below* baseline. GEMM autotuning
that changes output text is broken tuning.
The decode curve without spec decode is dead flat 12.11 tok/s from 304 to
16,095 tokens — decode is context-independent on this shape, bound per-step
by the expert gather, not by KV depth.
## Telemetry — the measurement that found the win
`R4D_LRU_TELEMETRY=1` arms per-layer counters; a torch-profiler window
(`/start_profile` → traffic → `/stop_profile`) makes the fork log and write
`expert_cache_rank<N>.json` beside the traces. **`torch_profiler_with_stack=false`
is mandatory in the profiler config** — the default (true) grew a 6.5-min
window's trace buffer ~40 GB and the kernel OOM-killed the engine core
(anon-rss 95.2 GB, `oom-kill` in `journalctl -k`). The short window at
166 ctx (~15 s of traffic) completed cleanly with `with_stack=false`.
The fork's own numbers (102 steps, both ranks identical):
```
hit rate 0.8245 (137,084/166,262) host installs 15,814
read-through 13,364 prefetched 36,720
served by prefetch (migrations) 13,369
```
- **The LRU is good**: 82.45% hit rate overall; layers 0-2 pinned at
100%; swap-region layers 78-81%. The "arena holds only 27% of experts
→ 73% miss" arithmetic was wrong — temporal reuse carries it.
- **The prefetcher is the poison**: 36,720 slabs moved to serve 13,369
(36% precision). The 23,351 unused copies are ~0.7 GB/step/rank of pure
waste — roughly half of all gather traffic — and the 13,369 "savings"
would have been installs anyway. Depth-1 prefetch also cannot cover the
~2,400-slab per-step working set with its 16 slots/layer (768 total).
- Per-layer-step at a swap layer: ~30.6 routed, 24.8 hits, 3.3 installs,
2.6 read-through, 8.0 prefetched of which 1.7-3.3 useful.
## The wall, measured
- **PCIe link (pcie-link-probe):** GPUs `03:00.0`/`07:00.0` LnkCap/LnkSta
32GT/s x16; root ports `00:01.1`/`00:01.3` cap x8. Effective **Gen5 x8 =
31.5 GB/s/direction**. `rocm-smi pcie_link_speed: 25` is the ASPM idle
downshift, not the loaded rate.
- **H2D (HIP probe, pinned):** 2.345 MiB (the engine's exact expert slab)
**22.74 GB/s**; 16 MiB 28.19; 1 GiB 28.74 (91% of ceiling). Pinned ≈
pageable — staging is not the lever. Single-transfer latency at slab size
105.7 µs vs 78 µs ideal → ~21% per-transfer overhead.
- **Engine's effective gather rate:** ~6.5-9.5 GB/s = **30-42% of what the
link delivers at the same size**. The bottleneck is the fork's read path
(chunk/lane geometry, per-transfer cost), not the wire.
- **Duty cycle:** peak GPU util 100%, smooth ~29% with prefetch on — each
step is ~100 ms of compute then hundreds of ms waiting on expert
streaming. rocm-smi reports *utilization*, which reads ~0 while weights
load during an arm's reload; VRAM is the honest panel there.
## What the remaining flags cannot do (all measured)
- **Gather geometry** — 16x32 and 32x64 measured no better despite the
fork's "tuned at the Gen4 ceiling" comment; the constants are already at
parity on Gen5 x8. The read-path gap is structural, not a tuning knob.
- **Spec depth** — 4 is the optimum both ways (with prefetch on). Depth 6
additionally proves the UVA read-through path (`step tokens >
R4D_LRU_UVA_TOKENS=5`) is *slower* than copy-in.
- **Prefetch anything** — depth 2 neutral, depth 1 the new negative; the
tier is 3x too small for the working set and the predictor is 36%
precise. `expert_prefetch_k/budget/slots` exist (defaults 20/14/16) but
the precision result makes tuning them pointless.
- **Arena growth via shrink** — `UTIL=0.99` fails boot (`Free memory on
device cuda:1 (31.29/31.86)`); `--max-num-seqs 2` fails CUDA-graph
capture (`Constraints violated (L['query_start_loc'].size()[0])`); and
the MNBT 8192 arm already proved a ~1 GiB arena shift is decode-neutral
in both directions.
- **Concurrency** — under TP=2 both streams' tokens are computed on both
cards, so each rank gathers the union of both streams' misses; the links
do not add capacity and the shared arena thrashes. Summed 2-way (25.49)
measured *below* single-stream (26.8) with prefetch on.
- **ngram-only drafting** — fails at engine init:
`RuntimeError: PLE inputs were not prepared`; the fork prepares this
model's PLE inputs only on the MTP spec path. Needs a fork patch, not a
config.
- **V-only q8 KV** — stock `--kv-cache-dtype` offers fp8 variants only
(K+V together); a V-only q8 needs a fork patch.
## Fork bugs / traps found (for the engine authors or a future patch)
1. `CLAV_TUNABLEOP_SWEEP=1` is silently skipped whenever expert offload is
active unless `CLAV_TUNABLEOP_WITH_OFFLOAD=1` is also set
(`tunableop_warmup.py`).
2. TunableOp-with-offload, once forced on: +6.7 min warmup, no persistent
cache (container-local), and corrupt numerics (repetition loops,
early-EOS halts, invalid measurements). Rejected outright.
3. The torch-profiler stop path hangs the workers on ROCm when
`torch_profiler_with_stack` is true (default): >10 min hangs on two
attempts while the AsyncLLM wrapper reports "stopped successfully" in
1 s; the default stack-tracing grew a 6.5-min window's buffer ~40 GB
and OOM-killed the engine core. With `with_stack=false` the stop
completes and the telemetry report runs. The telemetry counters remain
reachable ONLY through that profiler stop — no direct RPC trigger
exists (`telemetry_report` has exactly one caller, `gpu_worker.py`).
4. ngram spec method does not prepare PLE inputs (above).
## Where the headroom lives now
The 35 tok/s target is met (37.4-40.6) with prefetch off. What is left:
1. **Fork patch to the gather read path** — the ~6.5-9.5 → 22.7 GB/s
software gap at slab size. If the gather reached wire rate, decode
lands in the 60+ tok/s band before acceptance effects. Code work in
`expert_lru.py`'s gather (`R4D_LRU_GATHER_*` is read at import; the
copy-in path itself is compiled — see `/app/r4dhip/r4d.so`).
2. **256k ctx — measured, final**: boots at `MAXLEN=262144` with
`KV_BYTES=4831838208` or `4328521728` alike — a 262,144-token window
needs **4.0 GiB** KV (hybrid attention + mamba pages ≈ 16 KiB/token,
not the 32k plan's 9,190 B/token; 3 GiB boot-refuses and the engine
estimates max len 184,224 at that pool; the exact-4.0 override loses
~20 MB to accounting and boots as 3.98 < 4.0). Decode band
**34.1-36.5** across both KV sizes — the arena is **identical (5,648
slots)** at 4.5 and 4.03 GiB (the planner buckets; the trim went to
margin, not slots), so the window costs ~10% decode vs the 32k config
and no KV trim recovers it. The alternative that keeps the full
37.4-40.6 band is `MAXLEN=184224` at 3 GiB KV. Correction: the planner
sizes the arena AFTER the KV override (the earlier "before" reading was
wrong) — the arena is what remains.
3. **2+ slots at long ctx** — the KV pool post-spec holds ~66k tokens at
2 GiB (~2x 32k streams); concurrency summed *below* parity at 4064 ctx,
so slots are a capacity feature, not a throughput one. Two full 256k
streams need ~8-10 GiB of KV — that fits the arithmetic only by cutting
the arena below usefulness; slots-at-256k is a fork-patch or
smaller-payload question.
## Reproducing
Host-side scratch scripts (2026-09-29): `/home/dank/serve_qwen_fp8.sh` (the
serve script with all hooks), `/home/dank/mtp_measure.py` (usage-counter
decode measure, 3 depths), `/home/dank/mtp_measure_short.py` (single depth,
for profile windows), `/home/dank/bench2way.py` (concurrency), plus the arm
scripts `arm_*.sh` / `sweep_*.sh` and the detached runner `/home/dank/job.sh`
(outputs under `/home/dank/jobs/NAME.out|.rc|.done`). The measure scripts
are `podman cp`'d into the container after every recreate (the serve script
re-copies them; a recreate wipes container-local files).
## 2026-10-01 — production cutovers (qwen FP8+MTP4, then bf16 no-spec)
Two same-day cutovers moved this page's findings into production:
1. **Qwen FP8 + MTP4 replaced the failed GLM/DFlash2 experiment** (the GLM
window is documented in [[frank-archon-glm53-flash-w4a16]]; git tag
`frank-archon-fp8-mtp-20261001` is the last commit of that shape). The
production trio took MAXLEN 200000 x 3, fp8 KV, MTP4, prefetch off —
measured 37.4-40.6 tok/s at ~54% acceptance.
2. **The overnight redo campaign re-measured EVERY lever at matched
geometry** (prefetch OFF everywhere, MAXLEN 200000/SEQS 3/fp8 baseline)
and concluded the day: the incumbent cutover to **bf16 KV, no
speculative decode** (Frank's call).
### What the redo campaign measured (all at shallow depth, 166/1087/4064)
| lever | verdict |
|---|---|
| prefetch (re-confirmed) | the whole night's +40-51% lives in deleting it; the old 12.11 "no-spec baseline" was itself prefetch-taxed |
| gather geometry | **R4D_LRU_GATHER_CHUNKS=32 / LANES=64 = +18.7%** (49.25 @4064), the win GROWS with depth (39.5@166 -> 54.1@1087); g8x16 mild, g64x128 past peak |
| KV dtype | bf16 (`auto`) +4% decode on fp8-era numbers but halves attention-KV capacity (measured 14,047 vs 7,926 B/token; mamba states unaffected) |
| spec family | no-spec fastest once the tax was gone (41.46/41.29/41.47); MTP4's drafter reserves ~2.6 GiB/rank beside the pool; untaxed MTP re-measures never completed |
| MNBT | 512 doubles TTFT (2.46 s vs 1.36 s @4064); 1024-4096 interchangeable; 12288 -2.5% + arena; **locked 1024** |
| UVA | 2048 POISON (-39%); 8192 ~parity |
| arena | flat ±1.4% from ~6,830 down to ~5,500 slots; knee below |
| six slots | every SEQS-6 KV sizing refused at engine init |
Depth caveat: every number above is at a near-empty KV. Deep-context
behavior (the gather win growing with depth, bf16's decode cost at depth)
is only sampled at 3 depths; a 64k-192k-depth probe is the flagged gap.
### The deployed shape (2026-10-03 shape G: 3 slots x 200704, fp8, precise memory, MNBT 2048, arena at the flag ceiling)
```
FRANK_ARCHON_MAXLEN=200704 200704-token window
FRANK_ARCHON_SEQS=3 --max-num-seqs 3 (SEQS=2 CANNOT boot - below)
FRANK_ARCHON_KV_BYTES=3166500000 PER RANK (fp8, 7,892 B/token -> 400,700 tokens)
FRANK_ARCHON_KV_DTYPE=fp8 --kv-cache-dtype fp8 (quantized)
FRANK_ARCHON_MNBT=2048 --max-num-batched-tokens 2048 (arena-free, see shape F)
FRANK_ARCHON_UTIL=0.982 budget = total x util, the law (1.0 unbootable, see below)
FRANK_ARCHON_CUDAGRAPH_BASE_MB=0 / _PER_GRAPH_MB=12 -e VLLM_CUDAGRAPH_MEM_*
FRANK_ARCHON_PRECISE_MEMORY=1 --expert-precise-memory (overhead 7.60 -> 5.25 GiB)
FRANK_ARCHON_EXPERT_OFFLOAD_MEM=120 --expert-offload-mem (host-region cap, explicit)
FRANK_ARCHON_GPU_MAX_HW_QUEUES=2 -e GPU_MAX_HW_QUEUES=2 (HSA queues/GPU)
FRANK_ARCHON_CACHE_DIR=/home/dank/.cache/frank-archon mounted at /root/.cache
R4D_LRU_GATHER_CHUNKS=32 / R4D_LRU_GATHER_LANES=64 (container env)
NO --speculative-config, NO --expert-prefetch
FRANK_ARCHON_NO_PREFIX_CACHING=1 --no-enable-prefix-caching (2026-10-05, the QFN align-mode GDN state fault — below)
```
### 2026-10-05 — prefix caching OFF: the fork's QFN align-mode GDN state fault
**Symptom (Frank, 2026-10-04):** output turned repetitive/corrupted at
~40k tokens into a long conversation; short sessions stayed clean.
**Evidence (live `podman logs frank-archon-qwen`, the 2026-10-04 18:19
boot):** one 0.6 s burst at 18:46:07 PDT — 5.6 h into service, one
running request, and the engine's own 18:46:08 log line reading
"Prefix cache hit rate: 48.1%". 48 `gdn_hip` faults, then the
process-lifetime silence cap:
```
[gdn_hip] ERROR gdn_decode_r: state out of ssm-cache range: max|S|=0
with non-finite values (slot=42 head=3 limit=3.40282e+38) fault #1-23
[gdn_hip] ERROR gdn_decode_r: ... (slot=42 head=8 ...) fault #24-46
[gdn_hip] ERROR: state fault report limit (32) reached; further faults
are counted silently
```
One physical SSM slot (42), two heads (3 and 8 — one per TP rank), 23
GDN layers each, the state non-finite (NaN). The guard's own text in
this build: *"known Qwen3.8-Flash-Next (QFN) flaw under prefix caching
and is being actively worked; output quality for this request may
degrade. Mitigation: --no-enable-prefix-caching."*
**Mechanism.** With prefix caching on, the fork defaults QFN to
`mamba_cache_mode=align` (boot log: "Mamba cache mode is set to 'align'
for Qwen4ExpForConditionalGeneration … when prefix caching is enabled";
`config/cache.py`: "none" is the no-prefix-cache mode, "all" is
deprecated — align is the only sound mode WITH the flag). Align mode
checkpoints each sequence's GDN (Gated DeltaNet) recurrent state into a
mamba page per block and RE-SEEDS the state from that page when a
request resumes from a cached prefix. The block size is raised to
**3136 tokens** for the page-size equality (boot log: "Setting
attention block size to 3136 tokens to ensure that attention page size
is >= mamba page size" + "Padding mamba page size by 0.13% … exactly
equal"). The corrupt state landed at the **13th boundary: 13 × 3136 =
40,768 tokens** — the reported "~40k" mark. The NaN persisted in the
mamba page, so every later turn of that conversation re-seeded the same
non-finite state for those two heads (silently — the fault-report cap
was already spent) while fresh short sessions looked fine.
**RAM ruled out the same day** (the 2026-10-04 17:17 host reboot is when
the current memory speed took effect): zero MCE/EDAC kernel records
since boot; an 8 GiB in-band pattern check stable over 60 s under live
load; 4800 MT/s is the JEDEC DDR5 floor on 5600 MT/s-rated DIMMs (not an
overclock); and a two-head, single-slot NaN state is a state-path fault,
not scattered finite bit corruption.
**Fix.** `FRANK_ARCHON_NO_PREFIX_CACHING=1` → `--no-enable-prefix-caching`
(the fork-named mitigation; the flag renders in both directions from the
knob so a fork default change cannot silently flip the contract). Cost:
no prefix reuse — every turn re-prefills the whole history instead of at
most 3135 tail tokens. A silently corrupted conversation is worse than a
slower first token.
**Verified 2026-10-05 (deploy b820d34):**
- Engine argv carries `--no-enable-prefix-caching`; health 200; warm boot
`init engine … took 76.57 s` (the mounted compile cache is intact).
- Boot plan unchanged, as predicted: expert cache **17.77 GiB / 7,762
slots** (the flag touches no plan term), CUDA-graph estimate 0.07 GiB
vs 0.08 GiB captured (inside the 12 MiB/graph charge).
- `GPU KV cache size: 424,678 tokens` — **up** from 401,408 at the same
3,166,500,000 B ask: the pool now prices ~7,456 B/token (was 7,891.8),
the mamba/side per-token term dropping once the per-block checkpoint
pages are gone; the 3136 attention block-size raise nonetheless stays
(the page-size invariant is enforced in mode "none" too). Frank's
400,700-token floor is cleared with ~24k of headroom.
- **Zero `gdn_hip` lines** in the boot-and-traffic window (only the two
INFO module banners mention the name).
- Boundary probe (single request, no prefix reuse): prompt 39,280 +
completion 1,491 = **40,771 total tokens — across the 40,768 line**;
the task (write 1–400, one per line) completed coherently: 399 integers
parsed, 0.997 monotone, tail `… 398 399 400`. Pre-fix, this boundary is
exactly where the conversation degraded (the 13th 3136-page re-seed).
- e2e: 32/33 pass; the one failure is the admin spec's
192.168.0.112:8989 leg with FrankBook-Pro asleep (environmental,
unrelated to this change; re-run `tests/e2e/run.sh` once the mini is up).
**RESULT — REVERTED THE SAME DAY.** With every mechanical check green
(zero faults, coherent 40,771-token probe), live long conversations
STILL degraded at ~40k on the running endpoint (Frank, direct
observation). The container serving that window (`--no-enable-prefix-
caching`, booted 2026-10-05 12:47 PDT) logged ZERO `gdn_hip` faults
while the corruption reproduced — so this recurrence is **fault-free**:
the NaN-state guard did not fire, which the re-seed theory requires.
Open hypotheses, in order: (1) the poisoned transcript is being re-fed —
corrupted assistant turns already in the history keep producing
corrupted continuations even with perfect state, so the same
conversation cannot distinguish fixed from unfixed; (2) a second
long-context mechanism with no guard coverage — fp8 KV degradation at
depth is the next suspect (a bf16-KV boot costs ~2× the pool per token
and would settle it). Prefix caching is **back ON**
(`FRANK_ARCHON_NO_PREFIX_CACHING` commented out — the cost bought no
fix). The knob and the serve branch stay for a future fork build with a
fixed GDN path. Next diagnostic: reproduce on a FRESH conversation; if
it still breaks at ~40k with zero faults, A/B the KV dtype.
**RESOLVED (2026-10-05 evening): it was the RAM.** Frank dropped the
memory from 4800 to **4600 MT/s** (in effect since the 2026-10-05 12:21
PDT host boot) and the corruption stopped: zero `gdn_hip` ERROR lines in
the container since, zero MCE records, and the live agent session on
this endpoint past 140k context, "comfy cozy" (Frank). Marginal DDR5 on
four DIMMs is a common cause for BOTH eras: the 2026-10-04 NaN burst
(corrupted weight/activation traffic poisoning a GDN state write — the
guard-visible path) and the fault-free ~40k degradation (silent flips in
KV/expert data, which no guard covers on non-ECC memory). The re-seed
theory is therefore UNPROVEN, not disproven — the entire caching
experiment ran on marginal RAM and was confounded on both arms.
Post-mortem on the rule-out: consumer non-ECC RAM raises no MCE for a
silent flip; a 60 s / 8 GiB pattern sample covers ~0.5 % of 246 GiB;
"4800 is the JEDEC floor on 5600-rated DIMMs" is a rating, not a
stability proof at four-DIMM loading. The revert test was the decisive
instrument and it pointed at the RAM all along. Prefix caching stays ON
(zero faults under heavy re-send traffic since the speed drop); the knob
stays for a future fork fix. The fp8-KV A/B is NOT needed.
### Shape F/G (2026-10-03): where the free VRAM actually is, and what forcing it cost
`rocm-smi` showed **3.57 GiB free per R9700** at the deployed shape E (28.28 GiB
used of 31.86 GiB, after hours of real traffic — so it is the high-water mark,
not an idle artifact). Growing the expert arena into it is NOT a matter of
raising `--gpu-memory-utilization`, because the arena is
```
arena = budget - non-expert - KV reserve - CUDA graphs - runtime overhead
budget = device_total x gpu_memory_utilization # expert_plan.py:567
```
and every term on the right is either fixed or a lie:
| term | plan charges | measured reality | reachable from our side? |
|---|---|---|---|
| `budget` | 31.22 GiB (0.98) | field is `le=1` so 31.86 GiB is the field max, but 1.0 is UNBOOTABLE (below) | **yes** — shipped 0.982 |
| `non-expert weights` | 6.28 GiB | checkpoint census | no |
| `KV reserve` | 1.91 GiB (registry 9,190 B/token x 200704 +8%) | real pool bills 2.93 GiB at 7,892 B/token | no (registry constant) |
| `CUDA graphs` | 0.55 GiB = 256 MiB + 52 MiB x 6 | `Graph capturing finished in 3 secs, took 0.31 GiB` | **yes** — `VLLM_CUDAGRAPH_MEM_BASE_MB` / `_PER_GRAPH_MB`; `VLLM_CUDAGRAPH_MEM_ESTIMATE_MB` pins the whole term outright (checked first, pair ignored when set) |
| `runtime overhead` | 5.25 GiB (`--expert-precise-memory`) | far lower at idle | no (registry constant) |
Shape F took the graph term (0.55 -> 0.32 GiB) and nudged util to 0.982:
**arena 17.52 GiB / 7,653 slots measured**, +131 slots. Shape G takes the
graph term further (0.32 -> 0.07 GiB) at the same util.
**Shape G takes the graph term to the measured floor, util to the law.** The
graph base pool is the actual over-charge: shape F's own boot reported
`Graph capturing finished in 4 secs, took -0.06 GiB` — a **negative** capture
delta, so the 256 MiB base was fiction. Charging `0 base + 12 MiB/graph`
(0.07 GiB for the shipped 6-graph plan, and still per-graph so an 18-graph
spec-decode shape pays 0.21 GiB) lands the plan at budget 31.29 / graphs 0.07 /
**arena 17.77 GiB / 7,762 slots measured**. The KV ask went to
`3,166,500,000` B/rank, which the engine's own 7,891.8 B/token price lands at
**401,408 tokens measured / 2.00x concurrency** — the first shape holding two
full-length conversations with real spare. (The ask/landed delta is block
alignment: 196 x 2048. The price is shape-dependent, 7,701 B/token at 262144.)
**UTIL 1.00 is not a knife-edge on this fork, it is UNBOOTABLE — measured
2026-10-03.** One deploy took the field maximum to buy the last 0.57 GiB of
arena; both workers refused init BEFORE loading a single weight, twice:
`ValueError: Free memory on device cuda:N (31.29/31.86 GiB ...)` — the worker
demands free > budget and budget == total, so `le=1` on the CLI field is a
trap: the engine dies in a 2-restart crash loop with zero payload progress.
So `0.982` is a LAW, not a preference, and the "why 0.982" answer now has two
proofs: this ValueError, and the historical ceiling deaths (the 2x262144 bf16
boot at 0.13 GiB free; the 5x262144 boot at "0 bytes free" on a real
995-token request). Everything beyond ~7,912 slots needs the registry edited
inside the image (the fork's log line says exactly that: *"Update the
registry entry if this drifts"*), not a deploy.
**The host region was the last arena-plan term still autofitting per boot —
now it is a literal (2026-10-04).** With no `--expert-offload-mem` the planner
binds the host region to `MemAvailable` AT BOOT minus `PLE 47.68 - 2 x 7.0
process - 5.0 api/engine - 5% margin`: the 2026-10-05 boot read 237.97 GiB and
offered 159.39 GiB usable, while the same host reads **75.4 GiB available while
running** — the bound tracks whatever the co-tenants hold at that moment. The
demand is 105.48 GiB (52.74 GiB/rank x 2, 23,040 swap-target experts), and a
boot where the bound drops near it silently converts next-hottest experts to
VRAM-only (the `pinned_extra` warning is the only trace).
`FRANK_ARCHON_EXPERT_OFFLOAD_MEM=120` pins it: measured demand + 14% headroom,
on a host with 246.1 GiB total. Semantics verified in the shipped image
(`offloader/expert_plan.py`): the cap binds as the bound, an exceedance only
warns, and the plan rewrites the field to the cap it met (+1 MiB/rank).
**MNBT 1024 -> 2048 is free on every axis the plan can see.** The registry
overhead model is `base + 0.22 GiB x max(0, (nbt - 4096)/1024)` anchored at
nbt 4096, so any rung up to 4096 charges the identical 5.25 GiB; the fitted 5.25
was itself fitted at nbt 4096, which means 2048 is covered by an activation
figure measured at a **larger** batch. SEQS 3 boots at both 1024 and 2048
(measured 2026-10-02 — the `query_start_loc` `ConstraintViolationError` is a
SEQS property). The costs are a one-time recompile (the compile **range** is
part of the cache key, so `FRANK_ARCHON_CACHE_DIR` pays once) and a wider
prefill chunk, which the 09-29 sweep found decode-interchangeable across
1024-4096 while 512 doubled TTFT at 4064 ctx (2.46 s vs 1.36 s).
**`--expert-precise-memory` is the arena lever that does not touch the window.**
It attacks the planner's overhead term (shape F above attacks the graph term —
the third and last term this repo can reach), and the flag swaps the default
registry figure for the fitted one. Measured
2026-10-02 by a boot probe on the shape-D base (production stopped, candidate
booted AS the service; numbers are the engine's own plan lines):
| plan line | shape D | **shape E** |
|---|---|---|
| `runtime overhead (non-torch + buffers + activation)` | 7.60 GiB | **5.25 GiB** (`precise (gfx1201 TP2 nbt4096/seqs16/len262144)`) |
| `expert cache space` | 14.87 GiB | **17.22 GiB** |
| slots | 6,496 | **7,522** (+1,026, +16%) |
| `swap region` | - | 13.7 GiB / 5,986 slots, 133-134 per layer |
| boot | - | healthy in 420 s, no OOM |
`'expert_precise_memory': True` also appears in `non-default args`, so a silent
argparse drop is visible in the boot log. The flag lands **in the arena, not the
pool** - `--kv-cache-memory-bytes` is untouched at 3.15e9/rank (option A), because
the pool is not an arena lever and cutting it further would only cost
concurrency: a 2.2e9 pool was probed in the same window and reports 277,686
tokens / **1.38x**, so it is the conservative fallback if free VRAM ever binds,
not the shipped value.
**The compile cache is a boot-cost fix, and it was worth 173 s per boot.**
Measured 2026-10-02: the image ships **no `/root/.cache` at all** (a fresh
container: `No such file or directory`), and after a boot that directory holds
`vllm` at **1,154,714,333 B in 4,665 files** plus `comgr` at 229 MB, all inside
the container's ephemeral layer. Every
`podman rm -f` - i.e. every deploy, restart and reboot - discards it, and the
engine re-pays:
```
Using cache directory: /root/.cache/vllm/torch_compile_cache/b265bd396a/rank_0_0/backbone
Dynamo bytecode transform time: 4.99 s
Compiling a graph for compile range (1, 1024) takes 173.19 s
saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/…
torch.compile took 181.20 s in total
init engine (profile, create kv cache, warmup model) took 217.26 s (compilation: 181.20 s)
```
Mounting a host directory at `/root/.cache` makes every later boot load the AOT
artifact instead. The mount is deliberately the **parent**, not
`/root/.cache/vllm`: `VLLM_CACHE_ROOT` defaults to the `vllm` child, and
`comgr` is the HIP-kernel JIT cache, so one mount persists both — and it hides
nothing, because the image ships no `/root/.cache`. The cache key
includes the compile range, so the MNBT choice costs exactly one compilation.
Entries are version- and config-hashed: a stale entry is a miss, never a wrong
answer.
**Proven live on the deployed endpoint, 2026-10-02, consecutive boots of this
exact shape:**
| boot | engine log |
|---|---|
| cold (deploy) | `Compiling a graph for compile range (1, 1024) takes 179.24 s` → `torch.compile took 188.26 s in total` |
| warm (restart) | `Directly load the compiled graph(s) for compile range (1, 1024) from the cache, took 2.143 s` → `torch.compile took 3.97 s in total` → `init engine (profile, create kv cache, warmup model) took 41.13 s (compilation: 3.98 s)` |
The host directory held **1.4 GB in 4,707 files** after the cold boot. Time to
`/health` is dominated by the 172.8 GiB payload load (the warm boot reached
health in 490 s), so the win is the ~184 s of inductor/AOT build rather than a
proportional cut in boot time.
`GPU_MAX_HW_QUEUES=2` caps the HSA hardware queues per GPU. Verified a real knob
rather than a no-op - the literal is read out of
`/opt/rocm/core-10.0/lib/libhsa-runtime64.so` - and shipped deliberately
alongside the 32x64 gather geometry. It is **not** A/B measured.
**The pool, not the slot count, sets how many conversations live at once.**
Measured live at this shape: `GPU KV cache size: 398,658 tokens, Maximum
concurrency for 200,704 tokens per request: 1.99x` (3,146,129,408 B reserved).
The per-token price is **shape-dependent**: the 262144 shape priced at 7,701
B/token (6,100,000,000 B -> 792,069 tokens), this pool at 7,892 - ~2.5% dearer,
so a concurrency figure never extrapolates across geometries. The endpoint
advertises 3 slots: a third LONG request queues rather than being refused, and
shorter requests use the spare blocks. Verified: three concurrent 67-token
requests answered in parallel, ~4.1 s each. Two contexts at 200704 are a
capacity contract, not a throughput one (2-way concurrency summed BELOW parity
at 4064 ctx).
#### `--max-num-seqs 2` is impossible on this model (measured 2026-10-02)
Four single-variable boot probes against the then-deployed shape, production
stopped for each:
| MAXLEN | SEQS | MNBT | result |
|---|---|---|---|
| 262144 | 2 | 1024 | **Engine core initialization failed** |
| 262144 | 2 | 2048 | **Engine core initialization failed** |
| 262144 | 3 | 1024 | healthy - 792,069 tokens, 3.02x, arena 6,248 |
| 262144 | 3 | 2048 | healthy - the then-deployed incumbent |
The failure is `torch.fx.experimental.symbolic_shapes.ConstraintViolationError:
Constraints violated (L['query_start_loc'].size()[0]) ... specialized it to be
a constant (3)`, raised from AOT `build_guards(strict_error=True)` during
`determine_available_memory -> profile_run -> _dummy_run`. `model_runner.py`
sizes that input buffer `max_num_reqs + 1` and `_dummy_run` sets
`num_reqs = min(max_num_batched_tokens, max_num_reqs)`, so the width is
`SEQS + 1` for ANY `MNBT >= SEQS`: 3 at SEQS 2, 4 at SEQS 3. **MNBT is not a
lever here** - it only moves the compile RANGE ((1,1024) vs (1,2048)), which
is why the identical error appears at both rungs. `--enforce-eager` bypasses
the path at the cost of CUDA graphs. The same failure class killed the
2026-10-01 bf16 2 x 262144 shape.
#### The arena lever is `max_model_len`, not `--kv-cache-memory-bytes`
The fork's planner sizes expert cache as `budget - non-expert - KV reserve -
graphs - overhead`, and the KV reserve is registry-priced at **9190 B/token x
max_model_len** - not from the `--kv-cache-memory-bytes` value:
| MAXLEN | KV reserve | arena |
|---|---|---|
| 262144 | 2.48 GiB | 14.30 GiB / **6,248 slots** |
| 200704 | 1.91 GiB | 14.87 GiB / **6,496 slots** |
D(reserve) 0.57 GiB = D(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9
pool cut (2.75 GiB) bought **zero** arena slots. It buys runtime HEADROOM,
which is still worth having - the shape it replaced died with `0 bytes free`
on a real 995-token request. Grow the arena by lowering `max_model_len`, or by
taking the overhead term with `--expert-precise-memory` (below); the pool is the
headroom lever only.
#### The other arena lever: `--expert-precise-memory` (overhead 7.60 -> 5.25 GiB)
The flag is now ON. It was refused on 2026-10-01 for a reason that does not
generalize: the 2 x 262144 bf16 shape was already at the ceiling (weights 6.28 +
pool 6.80 + graphs 0.55 + arena 16.65 + non-torch ~1.45 = 31.73 of 31.86 GiB,
NRestarts=3 at ~6.7 min each), so the 2.35 GiB the flag moves into the arena was
the last stack over the line, not the cause. The plain bf16 2 x 262144 boot also
did not stay up: 0 bytes free at runtime, and a real client request (995-token
prompt, `max_tokens=261149`) killed the worker with `Tried to allocate 50.00 MiB
... 0 bytes free`.
At 200704 x 3 the flag fits with room and is worth **+1,026 slots (+16%)**. What
made it affordable is the window trim, not a pool cut: the arena is
DEVICE-resident, the pool is not part of the budget's arithmetic, and the 200704
window's KV reserve (1.91 GiB) plus a smaller attention footprint is what the
plan needed. The pool stays at 3.15e9/rank deliberately (option A) - cutting it
further would cost concurrency (2.2e9 -> 1.38x) and buy no slots.
Boot-probed 2026-10-02 (transient systemd units at the production uid +
memlock, full-log capture; the rootless bench probe had masked the production
memory environment). The night searched two concurrency contracts:
| rung | result |
|---|---|
| 3 x 262144, bf16, KV 11,047,010,304 | **OOM at graph capture** — `Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB already allocated by PyTorch |
| 3 x 245760, bf16 | **OOM identically** |
| 3 x 237568, bf16 | PASS — KV 718,606 tokens |
| 2 x 262144, bf16 + `--expert-precise-memory` | **DEPLOYED then crashed** — 0.13 GiB free at boot, NRestarts=3 (the ceiling, not the flag) |
| 2 x 262144, bf16 (no precise flag) | did not stay up either: 0 bytes free at runtime, worker killed by a real request |
| 3 x 200704, fp8, pool sized for 2 contexts | **DEPLOYED 2026-10-02 (shape D)**: 398,658 tokens = 1.99x concurrency, arena 14.87 GiB / 6,496 slots, the 2.75 GiB/rank pool cut returned as runtime headroom |
| **3 x 200704, fp8, + `--expert-precise-memory`** | shape E (2026-10-02): same 1.99x pool, overhead 5.25 GiB, arena 17.22 GiB / 7,522 slots, boot healthy in 420 s |
| 3 x 200704, fp8, MNBT 2048, graph term 0.32 GiB, util 0.982 | shape F (2026-10-03): arena 17.52 GiB / **7,653 slots** measured, budget 31.29, `Estimated CUDA graph memory: 0.32 GiB`, KV unchanged 398,658 |
| 3 x 200704, fp8, graph term 0.07 GiB, util **1.00**, KV 3.1665e9 | **DID NOT BOOT (2026-10-03)**: `ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` on both ranks, twice, zero weights loaded — the worker demands free > budget and budget == total. UTIL 1.0 is unbootable, full stop |
| **3 x 200704, fp8, graph term 0.07 GiB, util 0.982, KV forced to 400,700** | **DEPLOYED 2026-10-03 (shape G)**: arena **17.77 GiB / 7,762 slots** measured (+109 over F, +240 over E), KV **401,408 tokens / 2.00x** concurrency measured, capture 0.31/0.10 GiB vs the 0.07 charge (under-charged, boots and holds), driver used 30.97 GiB -> **0.89 GiB real free per card** held through a live 4,096-token generation (~38-39 tok/s single-request, +13% vs the 33.99 shape-E baseline); rollback is UTIL or KV_BYTES down a notch |
Six full-length contexts cannot fit at any KV dtype: the 6-seq graph/activation
footprint leaves only 9.18 GiB for the pool, and 6 × 262144 needs 12.47 GB even
at fp8.
The old "KV_BYTES is PER RANK, 9/16 GiB OOM" law was the MTP4 WORKSPACE
reservation, not a VRAM wall. Failures are clean and diagnostic: the engine
reports `torch.OutOfMemoryError: Tried to allocate X GiB. GPU N has ... of
which Y GiB is free`, and the fork's giant ERROR lines are split by journald —
read them via `journalctl -o json`, not `-o cat` (and `podman logs`, which is
only available while the container lives).
Rollback to the 2026-09-29 fp8+MTP4 shape: git tag
`frank-archon-fp8-mtp-20261001` (env trio + seam re-pinned there).
## Parallelism options on this box (2026-10-01 survey)
Surveyed from the running image's own CLI surface. Verdicts are specific to
dankserver: 2x R9700 (gfx1201), a ~172.8 GiB expert-offload MoE, and a model
with only 2 KV heads.
| option | flag | verdict on this box |
|---|---|---|
| TP | `--tensor-parallel-size` | **DEPLOYED, =2.** All-reduce dispatch shows `R4D_AR` then `PYNCCL` |
| DCP | `--decode-context-parallel-size` | **REJECTED at TP=2** — see below |
| PP | `--pipeline-parallel-size` | supported; untried (stage handoff plus a TP-keyed expert plan) |
| DP | `--data-parallel-size` (+`-rank`, `-start-rank`) | impossible: every replica holds a full model copy (172.8 GiB vs 31.86 GiB/card) |
| EP | `--enable-expert-parallel` + `--all2all-backend` (+`--enable-eplb`, `--enable-dbo`, `--enable-elastic-ep`) | supported (`naive`, `allgather_reducescatter`, `deepep_high_throughput`, `deepep_low_latency`, `pplx`, `mori`, `nixl`); would replace the fork's tiered-offload LOCAL gather with a per-layer PCIe all-to-all |
| multi-node | `--nnodes`, `--node-rank`, `--master-addr/port`, `--distributed-executor-backend` | supported; the actual path to more cards |
### DCP is structurally unavailable here — do not re-test on 2 GPUs
`--decode-context-parallel-size 2` fails `VllmConfig` validation at boot:
```
Decode context parallelism for GQA/MQA requires `--tensor-parallel-size` (2) to be
greater than the model's total number of KV heads (2). Increase
`--tensor-parallel-size` or set `--decode-context-parallel-size 1`.
```
DCP shards the KV along the *sequence* dimension after KV heads have been spread
across the TP ranks, so it requires **TP > num_kv_heads**. This model has 2 KV
heads (24 Q heads, GQA 12:1), so TP must be >= 4 — four GPUs, not two. Related
fork facts: DCP *reuses* the TP ranks rather than multiplying them (it is not
`TP x DCP`), and with PCP enabled the valid DCP sizes are `{1, pcp, tp x pcp}`
(`vllm/config/parallel.py`). The KV memory a DCP win would have bought is real —
each rank would hold half the sequence's KV — but it is unreachable without two
more cards.
### Measuring decode: the harness lives INSIDE the container
`/home/dank/mtp_measure.py` imports `/bench_iter.py` by absolute path, so it must
run in the engine's namespace and the files must be copied in first:
```
podman cp /home/dank/mtp_measure.py frank-archon-qwen:/
podman cp /home/dank/bench_iter.py frank-archon-qwen:/
podman exec frank-archon-qwen python3 /mtp_measure.py
```
Without the copy the exec dies with `can't open file '/mtp_measure.py': [Errno 2]
No such file or directory` — measured 2026-10-01, which silently cost a whole
arm's numbers. A window script must do the copy, not assume the path exists.
### The then-deployed shape, measured again (2026-10-01 A/B window)
A clean production-environment boot of the config deployed at that time (4 x 262144 fp8, MNBT
2048 in that window) reported **`GPU KV cache size: 1,079,582 tokens, Maximum
concurrency for 262,144 tokens per request: 4.12x`** — the same figures as the
deployed MNBT-1024 shape, i.e. MNBT does not move the pool, and the capacity is
reproducible across boots.
## See Also
- [[dankserver-frontier-moe-feasibility]] — the sibling feasibility page
(which frontier MoE fits this box; the same PCIe-expert-stream wall at
V4.1 scale)
- [[frank-archon-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (the R9V
fork era; production has been the tcclaviger fork since)
- [[x4-pair-experiment]] — the same residency law measured on a narrower
link: streaming over PCIe loses, computing in RAM does not care
- [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here
serverAppConfigs-docs/LLM/dankserver-vram-growth.md
+2 -2
View File
@@ -116,7 +116,7 @@ Verified 2026-08-14 after deploy: flash attention enabled, warmup clean, decode
## Mitigation Options
The documentation for the benchmark ([[frank-zealot-benchmark]])
The documentation for the benchmark ([[frank-zealot]])
prescribes the first option for this crash class. None of these change
the model.
@@ -135,6 +135,6 @@ PR #25649 (slot checkpoints freed on prompt clear, merged), issue #26119
## See Also
- [[frank-zealot-benchmark]]
- [[frank-zealot]]
- [[Dankserver System Specs]]
- [[deployment]]
serverAppConfigs-docs/LLM/dflash2-large-vulkan.md
-174
View File
@@ -1,174 +0,0 @@
---
type: reference
title: DFlash2 on Dankclaw (frank-dragoon)
description: ARCHIVED - unsloth Qwen3.8-27B-UD-Q8_K_XL served by upstream llama.cpp PR #27342 with a z-lab DFlash2 drafter on the Vulkan backend. Retired 2026-08-21 for speed; kept as a runbook.
tags:
- llama.cpp
- vulkan
- dankclaw
- gfx1151
- dflash
- dflash2
- speculative-decoding
timestamp: 2026-08-21
---
# DFlash2 on Dankclaw (frank-dragoon) — ARCHIVED
> **Archived 2026-08-21.** This DFlash2 Vulkan deployment was retired the
> same day it shipped because it was slower than the CIRU runtime it
> replaced. `frank-dragoon` is back on CIRU. This page stays as the
> runbook and provenance for the DFlash2 attempt.
`frank-dragoon` (port 52625) briefly served
`unsloth/Qwen3.8-27B-UD-Q8_K_XL` with a DFlash2 drafter from z-lab. The
runtime was upstream llama.cpp PR #27342 built with `GGML_VULKAN=ON`.
## Why Vulkan
DFlash2 selects its draft candidates with a lightweight path selector
built on the `TOP_K` and `ARGMAX` graph ops. The HIP backend does not
implement `TOP_K`. That is why the retired CIRU setup routed the MTP
draft around it with `LLAMA_MTP_CPU_ARGMAX_FASTPATH`.
The PR reference fork (Nathanw1014/strix-halo-llamacpp) validated DFlash2
on this exact gfx1151 through Vulkan (RADV). Its `test-backend-ops` passed
the new graph ops:
| Op | Cases |
|----|-------|
| TOP_K | 445 |
| CONCAT | 193 |
| PAD | 28 |
| REPEAT | 18 |
| FILL | 4 |
The whole stack (target + drafter) runs on `Vulkan0`.
## What Runs
| Artifact | Repo / File | Size |
|----------|-------------|------|
| Target | `unsloth/Qwen3.8-27B-GGUF` `Qwen3.8-27B-UD-Q8_K_XL.gguf` | 31.4 GB |
| Drafter | `z-lab/Qwen3.8-27B-DFlash2-GGUF` `Qwen3.8-27B-DFlash2-Q4_K_M.gguf` | 2.0 GB |
| Runtime | upstream llama.cpp PR #27342, GGML_VULKAN=ON | - |
The target was already on disk. The drafter was added for this deploy.
Both live in `/home/dank/models/`.
## Drafter Hyperparameters
Read from the drafter GGUF metadata:
| Parameter | Value |
|-----------|-------|
| `dflash.block_count` | 5 |
| `dflash.block_size` | 8 |
| `dflash.conv_kernel_size` | 2 |
| `dflash.conv_group_size` | 16 |
| `dflash.selector_rank` | 256 |
| `dflash.selector_top_k` | 16 |
| `dflash.context_length` | 262144 |
| `dflash.embedding_length` | 5120 |
| architecture | `dflash` |
The target and drafter share the Qwen3.8 architecture
(`qwen35`, dense 27B, vocab 248320), so the drafter inherits the target
tokenizer and LM head.
## Build Runbook (2026-08-21)
Performed on Dankclaw.
1. In `/home/dank/llama.cpp`, register the `ggml` remote and fetch the PR
head:
```
git -C /home/dank/llama.cpp remote add ggml https://github.com/ggml-org/llama.cpp.git
git -C /home/dank/llama.cpp fetch ggml pull/27342/head:pr-27342
```
2. Add a detached worktree that does not disturb the running build:
```
git -C /home/dank/llama.cpp worktree add --detach /home/dank/llama.cpp-dflash2 pr-27342
```
3. Configure with Vulkan only (glslang and `glslc` must be installed):
```
cmake -B /home/dank/llama.cpp-dflash2/build-dflash2 -G Ninja \
-DGGML_VULKAN=ON -DGGML_HIP=OFF -DGGML_CUDA=OFF -DGGML_METAL=OFF \
-DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release \
-DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_WEBUI=OFF
```
4. Build the server:
```
cmake --build /home/dank/llama.cpp-dflash2/build-dflash2 --target llama-server -j16
```
5. Verify the Vulkan device:
```
/home/dank/llama.cpp-dflash2/build-dflash2/bin/llama-server --list-devices
```
Output: `Vulkan0: AMD Radeon Graphics (RADV STRIX_HALO)`.
```mermaid
graph TD
A[Fetch PR #27342 head] --> B[Worktree on DFlash2 head]
B --> C[Configure Vulkan build]
C --> D[Build llama-server]
D --> E[Verify Vulkan0 device]
E --> F[Add DFlash2 drafter]
F --> G[Swap frank-dragoon]
G --> H[Health check + completion]
```
## Service
- Unit: `frank-dragoon.service`, port 52625
- Script: `scripts/serve-frank-dragoon.sh`, env `scripts/frank-dragoon.env`
- Binary: `/home/dank/llama.cpp-dflash2/build-dflash2/bin/llama-server`
### Launch profile
- 262144 context (native max), 1 slot, f16 KV (~16 GiB), 16 context
checkpoints, 8 GiB prompt cache
- DFlash2 draft, block size 8, device Vulkan0, drafter fully offloaded
- `--spec-type draft-dflash --spec-draft-n-max 8`
(clamped to block size 8 - 1 = 7 at runtime)
- `--temp 0.7 --top-p 0.95`, reasoning on (`--reasoning-format deepseek`)
- `--no-mmap`, flash attention `-fa on`, `--metrics`
### Required environment
No PromptForge variables are needed. The `TEMP`, `HOST`, and
`GPU_MAX_HEAP_SIZE` naming rule still applies for portability, even though
this build uses Vulkan, not HIP.
## Validation
The smoke test on 2026-08-21 used a scratch port (52626) and confirmed:
- Both models load and the server reports healthy.
- A chat completion returns reasoning and content.
- The DFlash2 drafter engaged:
`draft-dflash: #gen drafts = 11, #acc drafts = 10, #mean acc len = 5.27`.
- The per-position acceptance curve decays along the block
`(0.909, 0.818, 0.727, 0.545, 0.545, 0.364, 0.364)`, as expected.
The published mean acceptance for the Q4_K_M drafter is ~5.39, so 5.27
is close.
## Operational Notes
- Health: `curl http://127.0.0.1:52625/health`. Returns 503 while loading.
- Model load takes about a minute with a warm OS page cache (31 GB target)
and longer on a cold cache. Do not kill the process during warmup.
- `deploy.sh dankclaw` waits 20 minutes for health.
- The Vulkan device reports 113942 MiB. rocm-smi reports the 64 GiB
unified-memory budget. Trust the runtime for the addressable SVM pool.
- The MTP head of the target GGUF is ignored (`--spec-type draft-dflash`
replaces it). The load log warns about unused MTP tensors; this is normal.
## See Also
- [[ciru-activefpx-promptforge]] — the active CIRU runtime on Dankclaw
- [[DFlash DSpark Speculative Decoding]] — DFlash/DSpark design and history
- [[deployment]] — the deployment and update process
- [[qwen3.8-strix-optimizations]] — llama.cpp Optimizations on this iGPU
- [[index]] — knowledge base index
serverAppConfigs-docs/LLM/eemin-rocmi4-mtp.md
-188
View File
@@ -1,188 +0,0 @@
---
type: reference
title: caf-eemin Qwen3.8-27B ROCmI4 embedded MTP on Dankclaw
description: caf-eemin/Qwen3.8-27B-ROCmI4-MTP-GGUF (Q4_0_ROCMI4, embedded NextN/MTP head) served by the charlie12345/ROCmFPX llama.cpp runtime (ROCmI4 W4A4) on the Strix Halo iGPU. frank-dragoon.
tags:
- llama.cpp
- rocm
- dankclaw
- gfx1151
- rocmi4
- mtp
- iu4
timestamp: 2026-08-25
---
# caf-eemin Qwen3.8-27B ROCmI4 embedded MTP on Dankclaw
> **RETIRED 2026-08-25.** `frank-dragoon` served this model from 2026-08-18
> until 2026-08-25. It replaced the kingjones777 ROCmFP4 STRIX MTP payload
> on the same port and the same Strix Halo iGPU, running the new
> charlie12345/ROCmFPX ROCmI4 W4A4 build (see [[kingjones-strix-mtp]] for
> the page it replaced), and was itself replaced by
> [[kingjones-strix-rocmfp4]].
`frank-dragoon` (port 7770) serves
`cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF` (published under the public
`eemin` namespace) from the **charlie12345/ROCmFPX** llama.cpp fork built with
the opt-in `ROCmI4 W4A4` path. The model is `Q4_0_ROCMI4`: signed four-bit
weights packed two per byte, quantized from the BF16 Qwen3.8-27B and carrying
its **embedded NextN/MTP head** in the main GGUF — so there is no separate
draft file. The native gfx1151 `v_wmma_i32_16x16x16_iu4` matrix instruction
batch-verifies the MTP proposals for higher decode throughput.
## What This Is
**ROCmI4** is the GGUF weight format (signed 4-bit codes). **IU4** is the AMD
WMMA instruction spelling that consumes them. **W4A4** means four-bit weights
and four-bit activations during the accelerated matrix multiply. Ordinary
one-token decode stays memory-bandwidth-bound (MMVQ); batched prompt processing
and batched MTP target verification use the IU4 MMQ path. W4A4 is a lossy
activation-quant speed path — treat it as a speed/quality tradeoff.
The model is a conversion/quantization of `Qwen/Qwen3.8-27B` (Unsloth lineage,
Apache-2.0), not a retrained model. It reports architecture `qwen35`
(M-RoPE), ~27.32B params, native 262144 context, 866 tensors, ftype 108.
## Model Files
| File | Size | SHA-256 |
|------|------|---------|
| `Qwen3.8-27B-Q4_0_ROCMI4.gguf` | 13.50 GiB (14,534,384,384 B) | `ad51ecbee55930ababc629c377943a7bbe50f8aafcf32340c7cdae6fa6b7bb3c` |
| `mmproj-BF16.gguf` (vision, optional) | 0.87 GiB (931,146,432 B) | `83ee4f4f205fa514161778c41df1ea14144faa0f713510893b63c2395f5c2d53` |
Store under `/home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP/`. The vision
projector is optional for text-only use.
## Runtime Build
The stock llama.cpp build does not understand `Q4_0_ROCMI4` or the qualified
IU4/MTP path. Use charlie12345/ROCmFPX at the pinned commit, built HIP-only
(so the combined HIP/Vulkan scheduler cannot route the ROCmI4 tensors to
Vulkan, which has no shader for this type) with the W4A4 option on.
| Artifact | Path |
|----------|------|
| Fork + pinned commit | `/home/dank/rocmi4-qwen38/ROCmFPX` @ `c49ebdbd5c9f01ec242369f9e7f7967855f80cba` |
| Build | `cmake -S . -B build-rocmi4-w4a4 -G Ninja -DGGML_HIP=ON -DGGML_VULKAN=OFF -DCMAKE_HIP_ARCHITECTURES=gfx1151 -DGGML_HIP_ROCMI4_W4A4=ON -DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_WEBUI=OFF` |
| Binary | `/home/dank/rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server` (version 257, `c49ebdb`) |
| ROCm | system `/opt/rocm-7.2.4`, amdclang 22 |
Build a separate directory so the exact-int8 path (the default branch) stays
available. Rollback: disable `-DGGML_HIP_ROCMI4_W4A4=ON` and rebuild — no model
conversion or data change needed. The build prints
`ROCmI4 W4A4: enabled for device 0` when the accelerated path is active.
## Model Download
Downloaded from the public
`eemin/Qwen3.8-27B-ROCmI4-MTP-GGUF` repo (the card's download section still
names the still-private `cafonz/…`; use `eemin`). Verify against the repo's
`checksums.sha256` after download:
```
cd /home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP
sha256sum -c checksums.sha256
```
## Launch Profile
The qualified MTP profile (from the model card and PR #94, verified on the
live service): strict Qwen MTP, backend sampling, `n_max 4`, `p_min 0.60`.
```mermaid
graph TD
A[llama-server charlie12345 ROCmFPX c49ebdb<br/>ROCmI4 W4A4 enabled] --> B[load Q4_0_ROCMI4 GGUF + mmproj<br/>-ngl 999 -dev ROCm0]
B --> C[draft-mtp on ROCm0<br/>--spec-draft-ngl all]
C --> D[strict Qwen MTP<br/>n_max 4 / p_min 0.60<br/>backend sampling]
D --> E[Sharp chat template<br/>reasoning deepseek]
```
Key flags (see `scripts/serve-frank-dragoon.sh`):
```
--spec-type draft-mtp --spec-mtp-strict-qwen
--spec-draft-device ROCm0 --spec-draft-ngl all
--spec-draft-type-k f16 --spec-draft-type-v f16
--spec-draft-n-max 4 --spec-draft-n-min 0
--spec-draft-p-min 0.60 --spec-draft-backend-sampling
```
Because the MTP head is **embedded**, no `--model-draft` is passed and the
draft uses all layers on ROCm0. `--spec-mtp-strict-qwen` preserves the target
path's greedy decisions; it does not remove the W4A4 activation approximation.
### Serving settings
- 65536 context (192K below the native 262144 maximum), 1 slot, f16 KV,
context checkpoints, prompt cache. The lower limit leaves headroom for the
model plus KV cache in the host's unified-memory budget. Concurrent requests
queue; the server never preempts a running generation.
- `-t 16 -tb 32`, batch 512 / ubatch 256 (the qualified W4A4 profile).
- `--temp 0.7 --top-p 0.95`, reasoning on (`deepseek` puts `<think>` into
`message.reasoning_content`).
- Qwen-Sharp chat template (`--chat-template-file`), `enable_thinking` +
`preserve_thinking` on, preserving the peg-native + deepseek shape the Pi
agent expects.
- Strix Halo unified memory + native gfx1151 detection: `HSA_OVERRIDE_GFX_VERSION=11.5.1`,
`GGML_HIP_ENABLE_UNIFIED_MEMORY=1`.
### Service
- Unit: `frank-dragoon.service`, port 7770
- Script: `scripts/serve-frank-dragoon.sh`, env `scripts/frank-dragoon.env`
## Tuning MTP
MTP gains are content-dependent: predictable code, structured data, and lists
accept longer drafts than creative prose. The active conservative profile is
`n4 / p0.60`; the previously qualified starting point was `n16 / p0.60`.
## ubatch Benchmark
Decode throughput depends on the ubatch (per-eval token batch). Run the sweep
on 8k contexts, 512 → 4096 in steps of 512, with a fixed batch:
```
sudo systemctl stop frank-dragoon
./scripts/bench-ubatch.sh
sudo systemctl start frank-dragoon
```
The script stops `frank-dragoon` for a clean GPU, boots one server
instance per ubatch on a scratch port, reads authoritative `print_timing` tok/s,
and restarts the service. See [[frank-dragoon-benchmark]].
## Known Issues
1. **gfx1151 only** for the W4A4 path. Other GPUs fall back to exact int8 MMQ.
2. **Memory pressure:** the model is 13.5 GiB. Under `--no-mmap` with full
offload it needs the whole tensor set at once; leave room in the 64 GiB
unified pool (rocm-smi figure). Do not run a bench with the live service up
or it OOMs at load.
3. **`Q4_0_ROCMI4` + MTP flags need the ROCmFPX fork**; stock llama.cpp may
reject the file or drop the optimized path.
4. **W4A4 is lossy** (+~5.4% perplexity vs exact int8). Opt-in speed path.
## Measured Results (qualification)
From the model card, gfx1151, pinned build, W4A4 + strict MTP-16:
| Result | Value |
|--------|-------|
| HumanEval pilot, W4A4 IU4 (mean) | 49.40 tok/s |
| Full HumanEval, W4A4 (mean / median) | 44.39 / 45.23 tok/s |
| HumanEval / HumanEval+ pass@1 | 94.5% / 91.5% |
| Non-speculative tg128 | ~13.8 tok/s |
Local smoke test 2026-08-25 on the iGPU: model loaded, strict-MTP active,
draft acceptance 0.74 (mean 7.67), generation ~25 tok/s at a small output.
The ubatch sweep above gives the production ubatch choice.
## See Also
- [[deployment]] — deploy and update process
- [[kingjones-strix-mtp]] — the retired ROCmFP4 predecessor on this slot
- [[ciru-activefpx-promptforge]] — the retired v3 CIRU PromptForge page
- [[Dankclaw System Specs]] — the box
- [[frank-dragoon-benchmark]] — backend/spec benchmark history
serverAppConfigs-docs/LLM/frank-archon-freetoken-pr587.md
-184
View File
@@ -1,184 +0,0 @@
# frank-archon — FreeToken PR #587 ROCm (Qwen3.8-Flash-Next-Uncensored-FP8, 1× R9700)
**Status: RETIRED 2026-10-05 → (superseded the same day by the radiance
cutover — see [[frank-archon-radiance-qwen38-native-fp8]], itself retired
2026-10-08; the payload live today (2026-10-18) is the publisher iq4r container,
[[frank-archon-radiance-qwen38-iq4r]]). Kept as the rollback runbook: the image
is still in dank's rootless storage, the orcarouter payload is not on disk any
more)** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 ·
**GPU:** 1× Radeon AI PRO R9700 (gfx1201) — the second R9700 sits idle
The tcclaviger RDNA4 vLLM fork was replaced by **FreeToken at PR #587 head
`77a7201f98f00d93f618c6cf8b81ffe8360e0899`** (branch `rocm714_torch214`), built
on the host into the local image `localhost/frank-primary-freetoken:pr587-77a7201`
(that is the tag podman reports today — the 2026-10-10 rename never retagged it)
by `scripts/docker/frank-primary-freetoken.Dockerfile`, which survives only in
the rollback tag: `scripts/docker/` is not in the tree today. Same port, same
served alias (the wire id was `frank-primary` until the 2026-10-09 rename), same
131-shard payload (`/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8`, since
deleted from the host). Rollback:
git tag `frank-archon-vllm-shapeG-20261003` (the vLLM trio + its env lore),
runbook [[dankserver-qwen38-fp8-expert-offload-tuning]].
## Why FreeToken
The vLLM fork planned a per-boot VRAM/host tiering from an estimator: util
0.982 was a hard law (1.00 was **unbootable** — worker init raises
`ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` when budget ==
total), the expert arena capped at 17.77 GiB / 7,762 slots, and live traffic
still degraded ~40k tokens in (the GDN state-fault class, never fixed in the
fork). FreeToken keeps **all expert weights in pinned host RAM** with an LRU
GPU cache, sizes **KV as an explicit token pool**, and boots in one
deterministic pass — no per-boot estimator to fight.
## The build (on dankserver, rootless podman as dank)
```bash
git checkout frank-archon-freetoken-pr587-20261005 # scripts/docker/ exists only in the tag
cd /home/dank/serverAppConfigs
podman build -t frank-primary-freetoken:pr587-77a7201 \
-f scripts/docker/frank-primary-freetoken.Dockerfile scripts/docker/
```
(The names above are the ones the tag actually carries —
`scripts/docker/frank-primary-freetoken.Dockerfile`, per
`git ls-tree frank-archon-freetoken-pr587-20261005`. The 2026-10-10 rename
swept the docs, not the tag or the image in storage.)
Laws baked into the Dockerfile (each one cost a failure):
- **Base digest-pinned**: `rocm/pytorch:rocm7.14_ubuntu24.04_py3.12_pytorch_release_2.11.0@sha256:a223aee1…` —
the PR doc's known-good stack (torch 2.11.0+rocm7.14, triton 3.7). No
rocm7.14+torch2.14 image exists on Docker Hub (the 2.14 images are
rocm10.1.0 — an untested pair for this PR).
- **`--no-build-isolation --no-deps -e .`** — a plain
`pip install -e .` on ROCm can REPLACE the image's ROCm torch with a CUDA
build (the dependency set is authored for NVIDIA). Runtime deps are
curated explicitly; `flashlib==0.3.0 --no-deps` (its numba /
nvidia-cutlass-dsl deps are CUDA-only); ninja for the QSA JIT;
**flashinfer / sgl_kernel intentionally absent** (no ROCm builds;
FreeToken falls back to its Triton kernels — the PR's 12 pytest failures
are exactly those backends).
- **`libhiprtc.so` dev symlink**: the pip-ROCm torch ships only the
versioned `libhiprtc.so.7` in
`/opt/venv/.../site-packages/_rocm_sdk_core/lib`, but the PR's `_cpu_moe`
extension links `-lhiprtc`. First build died `ld: cannot find -lhiprtc`;
the Dockerfile symlinks the unversioned name into the ALREADY-LINKED dir.
- FreeToken source is checked out at the **pinned PR head**, not the branch
(a future push to `rocm714_torch214` cannot change the deployed engine).
- The stale Docker Hub login (`docker login` creds at
`~/.docker/config.json`) poisons podman's ANONYMOUS pulls of public images
with "invalid username/password" — `docker logout` fixes the pull. Hit
2026-10-05 while pulling the base image.
## Measured at cutover (2026-10-05/06)
- **Boot**: ~2.3 min container-start → API-ready (131 shards page-in + PLE
pin + expert staging + CUDA graphs `[1,2]`). VRAM after graphs:
**1.80 GiB free** of 31.86. RAM: 122 GB used + **113 GB shared (the
pinned expert/PLE slabs)**, 123 GB still available.
- **Decode: 17.1 tok/s** single-stream, temp 0, 400-token probe (clean
1→N counting). The vLLM incumbent measured ~30 tok/s on TWO cards;
FreeToken single-card is the deployed baseline.
- **The single-card LRU ceiling**: the KV contract (harness worst case
177,356 tokens) + dense fp8 + the vision tower + graphs leave ~12 GiB for
the expert LRU at rate 0.10 — and only 1.8 GiB of live headroom, so
`ft ctl cache --moe` cannot raise it far on one card. The paths to a
better decode rate are TP=2 (below) or a smaller pool (contract
violation — the 177,356 floor is real traffic, not padding).
- **TP=2 exists but does not boot on ROCm** (probed 2026-10-06,
zero-impact window): `--tensor-parallel-size 2 --gpu 0,1` dies in the TP
allreduce JIT — `pynccl.cu` compiles per-rank at boot and clang can't
resolve `cuda_runtime.h` from the hipified include path
(`1 error generated when compiling for gfx1201`, ninja stopped). The PR
author validated single-GPU only; this is FreeToken's untested-TP-on-ROCm
edge, not a board or driver fault (radiance TP=2 allreduces fine on this
pair since the 2026-09-20 swap). A future fix = patch the JIT include
path (HIP's cuda_runtime shim) or vendor an ROCm allreduce — re-probe
then.
- **Verification gates passed**: /health ok; plain completion `FLEET-OK`
(finish stop); thinking split — `reasoning_content` non-empty AND
`content` carries the answer (17×24 → "408"); native `tool_calls` with
correct name+arguments (finish tool_calls); 256×256 solid-red PNG through
an `image_url` part answers exactly "Red" (prompt_tokens 127); stats:
native ctx 262,144, KV 4,687 pages × 64 = 299,968, modalities
[text,image], sampling defaults from generation_config.json
(temp 1.0 / top_p 0.95 / top_k 20).
- **Thinking counts against max_tokens** (same as the vLLM era): a
`max_tokens: 10` probe returns EMPTY content with the budget spent in the
think block. Clients must budget for thinking or disable it.
- **Prometheus scrapes /metrics → 404** (Dankclaw scrapes 192.168.0.235:7770
from the llm-fleet job; FreeToken has no /metrics endpoint). The target
will read down until the scrape config is repointed or a metrics shim
lands — same class as the retired HTML-probe targets
(see [[Observability]]).
## The shape
| Knob | Value | Why |
|---|---|---|
| `--gpu` / `HIP_VISIBLE_DEVICES` | 0 / 0 | ONE R9700; visibility pinned so in-container index 0 is the right card |
| `--num-tokens` | **299,968** | KV pool in TOKENS; **must be a multiple of 64** (the auto attention backend is `qsa_sparse`, which overrides page size to 64 — 300,000 was refused: `nearest valid values: 299968 or 300032`) |
| `--max-running-requests` | 2 | first-boot conservative; raise after real-traffic slot-occupancy look |
| `--max-output-tokens` | 36,864 | harness maxTokens; requests that omit max_tokens get this default |
| `--moe-cache-rate` | 0.10 | GPU expert LRU as fraction of all experts (~120 GiB fp8 → ~12 GiB VRAM); first-boot conservative |
| parsers | AUTO | thinking → `reasoning_content` natively; tools in the qwen3.8 dialect; NO `--reasoning-parser`/`--tool-call-parser` flags |
KV math: 12 full-attention layers × 2 KV heads × 256 head_dim × 2 (K+V) ×
2 B = **24 KiB/token** (f16 KV; the 36 GDN layers use fixed per-seq state,
not per-token KV) → 299,968 × 24 KiB ≈ 7.0 GiB. Clears the harness worst
case (140,492-token prompt + 36,864 max_tokens = 177,356) and admits ONE
native 262,144-token conversation + full budget. `models.yml`
`contextWindow: 200704` stays (compacts early, never 400s).
## RAM math (why the root unit + memlock)
246 GiB total. FreeToken pins ~120 GiB of fp8 expert rows + the ~47.7 GiB
PLE ngram table in host RAM → the unit carries
`LimitMEMLOCK=infinity` (root unit; dank's login session caps at 8 MiB and
a user quadlet cannot raise it). Weights+PLE+overhead ≈ 190 GiB — the vLLM
container's ~170 GiB MUST be down first; `systemctl stop frank-archon`
frees it (measured: 207 GB free right after the stop).
## Resize live
```bash
ft ctl cache --kv <tokens> # grows the KV pool LIVE (shrink needs restart)
ft ctl cache --moe <rate> # same for the expert LRU
ft ctl stats # pool occupancy, VRAM, throughput
```
## Verification (the fleet's real-completion law)
1. `/health` 200 — necessary, not sufficient.
2. Plain completion returns the asked-for string.
3. Thinking request: `reasoning_content` non-empty AND `content` carries the
answer (the models.yml `reasoningContentField: reasoning_content` flip
rides on this).
4. Vision probe: a solid-color PNG through an `image_url` part answers the
exact color (the legacy top-level `images` field trap does not exist
here — FreeToken is OpenAI-native).
5. Unit-managed: `systemctl show -p MainPID` behind the health answer (the
2026-09-26 stray-port false green).
## Rollback
```bash
git checkout frank-archon-vllm-shapeG-20261003 -- \
scripts/frank-archon.env scripts/serve-frank-archon.sh \
scripts/frank-archon.service
./tests/run.sh && git commit -m "frank-archon: rollback to vLLM shape G" \
&& git push && ./scripts/deploy.sh dankserver --service frank-archon
```
The tcclaviger image and the FreeToken image coexist in dank's rootless
storage; the deploy re-pulls the pinned digest if absent.
## Index
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the retired vLLM shape
(rollback target; its 362-line env lore lives in the tag)
- [[freetoken-frank-small]] — the v0.1.2 CUDA precedent (ortobox, RTX 3090)
- [[LLM/index]] — fleet index
serverAppConfigs-docs/LLM/frank-archon-glm53-flash-w4a16.md
-808
View File
@@ -1,808 +0,0 @@
---
type: runbook
title: frank-archon — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
description: VERIFIED 2026-09-30 by boot probe, bench, live expert-cache telemetry AND a torch-profiler kernel-time window. frank-archon cut over from Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 to Intel/GLM-5.3-Flash-W4A16-AutoRound, the first payload this fleet serves through the fork's W4A16 expert-offload path (INC scheme -> R4dW4a16MoEMethod, libr4d grouped W4A16 GEMM). Measured decode 9.9-10.0 tok/s and prefill 918 tok/s at 131,072 ctx. THE 35 tok/s TARGET IS NOT MET, and the profiler window FALSIFIED the first explanation: the expert gather is 1.05 % of device time, not the bound. 44 % of the 93 ms/token budget is pointwise glue (1,428 elementwise launches per token) because the fork's GLM trunk is NOT torch.compiled while its own Qwen trunk is; the second-largest lever, the fused router, is binary-blocked at 288 experts. The compile changelist is staged but incomplete (three patches, one blocker left).
tags:
- dankserver
- r9700
- gfx1201
- glm5.3
- w4a16
- int4
- expert-offload
- expert-cache
- autoround
- vllm
timestamp: 2026-09-30
---
# frank-archon — GLM-5.3-Flash W4A16 int4 (tcclaviger, 2x R9700)
> **RETIRED 2026-10-01** (era 2026-09-30 → 10-01, matching the AGENTS.md Runbook
> Index). Cutover reversed to Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4 — the
> trio as of commit `2339fae`, restored by `d91379c` (2026-09-30). frank-archon
> runs [[frank-archon-radiance-qwen38-iq4r]] (radiance) today, 2026-10-18. The
> DFlash2 drafter experiment is a **failure**: after five patches the drafter
> served and drafted (12.8 tok/s proposed, 65,536 ctx, both TP ranks capturing
> FULL dflash2 cuda graphs) but **acceptance pinned at 1.00-1.38 on every text
> class including verbatim repetition** — the signature of a mis-wired
> aux-hidden-state capture, not a weak drafter. At ~1.2 acceptance the 8-token
> verification tax makes decode 5x SLOWER than the 9.9-10.0 no-draft baseline
> (1.6-1.8 tok/s measured). The untested alternative capture semantics: post-layer
> output state (DeepSeek's `hidden_states + residual`) or the mHC `comb` stream.
> Payload (170 GiB) + drafter (2.2 GiB) were staged to
> `/home/dank/.AGENTTRASH/20261001-glm-purge/` and purged since — on 2026-10-18
> Dankserver has no GLM tree under `/home/dank/models/` and `~/.AGENTTRASH` is
> empty. KV-cache work that survives: `glm53-dflash-04-kv-groups-draft.patch` and
> `glm53-dflash-04b-kv-registry-dflash.patch` are still in `scripts/patches/`;
> patch 03 (tailpin) was deleted at the rollback (`d91379c`) and patch 02 was
> never committed. None of these apply byte-exact today — the 2026-10-10 rename
> (`55f87fc`) rewrote the retired service names inside their context lines.
> **Question this page answers.** Can the fleet serve the 320B-class
> `GLM-5.3-Flash` at int4 on two R9700s with expert offload, and how fast is it?
> **Answer.** Yes — it serves, with vision, tool calls and 131,072 ctx. It runs
> at **9.9-10.0 tok/s decode / 918 tok/s prefill**, and **35 tok/s is not
> reachable** on this payload as the fork ships it. The bound is **not** the
> model, the quant, the link, or the expert gather (measured: 1.05 % of device
> time). It is the **unfused pointwise glue of an uncompiled trunk** — 44 % of a
> 93 ms per-token budget spread over ~1,428 elementwise launches per token —
> because the fork compiles its own Qwen trunk and not its GLM one. See
> [The measured profile](#the-measured-profile-and-the-real-wall).
## DFlash2 target-side port — the four laws and the KV ladder
Distilled 2026-10-05 from the probe windows (skills `glm53-dflash-boot-ladder`
/ `glm53-dflash-eagle3-port`); each law was proven by a real failed boot.
**The four EAGLE3-port laws** (make the GLM trunk satisfy `SupportsEagle3`):
1. **Patch the instance the runner calls.** `model_runner.py:425` calls
`set_eagle3_aux_hidden_state_layers(self.model)` on the *loaded* model —
GLM loads as `Glm5NextForConditionalGeneration` (multimodal wrapper;
`self.language_model = Glm5NextForCausalLM`). Members placed only on the
CausalLM fail the check on the conditional — same error text, wrong file.
Patch BOTH wrappers.
2. **Presence-checked member set.** `SupportsEagle3` (interfaces.py:1659) is
`@runtime_checkable`; `isinstance` walks ALL `__protocol_attrs__`
(including base-protocol members) with `getattr_static`, breaking on the
first miss. Both wrappers need, as class attributes:
`supports_eagle3: ClassVar[Literal[True]] = True`,
`has_own_lm_head: bool = False`, `has_own_embed_tokens: bool = False`.
Verify in-image with stock typing — no boot needed.
3. **MRO forbids inheriting the Protocol.** GLM wrappers carry four/five
Protocol-derived bases; adding `SupportsEagle3` anywhere dies with
"Cannot create a consistent method resolution order" (sim-proven: the
ordering rejects it in every base position). The attribute route avoids
the MRO, so the two methods must be written by hand: the CausalLM
unwraps to `self.model` (the `EagleModelMixin` trunk), the conditional
delegates to `self.language_model` (pixtral.py:498 pattern);
`get_eagle3_default_aux_hidden_state_layers` returns `(2, L//2, L-3)`,
with the drafter config's `target_layer_ids` winning when present.
4. **Trunk capture point:** append `aux_hidden_states` BEFORE the layer call
— GLM's loop carries the single-stream state entering the layer.
**The KV blocker ladder** (patches `scripts/patches/glm53-dflash-0*`):
1. EAGLE3 gate → presence-checked attrs on both wrappers (above).
2. KV page-size unifier: MLA layers cannot be padded (sparse MLA reads
whole token rows) but CAN grow block_size when the max page is an exact
multiple — bump `max_page_size` in `unify_kv_cache_spec_page_size` to
the `lcm` of all MLA-family pages first (patch 02).
3. kpool tail assert `tail_kv_cache.shape[2] == pool_size`: the unifier's
divisible branch grew KpoolTailSpec's block; its kernel pins block ==
index_kpool. Add a KpoolTailSpec branch that pads `page_size_padded`
instead of growing (patch 03).
4. KV capacity: the glm5 layout ALIASES mamba states INTO MLA pages (why
GLM fits 32 GiB cards); the generic path (taken when draft/hidden specs
exist) cannot alias → ~30 GiB demanded vs a 0.92 GiB reserve. Extend
`_get_kv_cache_groups_glm5_next` to partition
HiddenStateCacheSpec + drafter attention specs; the drafter forms its
own uniform group (per-group pages, NO unification → the MLA block
stays at the engine-auto-set 2,432 from interface.py:918, kpool guard
`block_size % 512 == 0` unstressed); layout returns a 10-tuple
(+draft_group, +hidden_groups); the builder sizes ONE shared backing
allocation (`allocate_kv_cache` asserts all KVCacheTensor sizes equal) =
aliased pool + appended draft/hidden tensors (patch 04). After 04 the
capacity requirement collapsed 31.76 → 1.88 GiB at 65,536 max len and
the engine SERVED (READY ~500 s, dflash2 fused prep ON, drafter FULL
graphs on both TP ranks).
5. Reserve registry: `_glm53_kv_bytes_per_token` must add 4,096 (drafter
KV) + 5×hidden×2 (aux hidden cache) keyed on
`speculative_config.method in {dflash, dspark, extract_hidden_states}`.
NEVER key on `use_aux_hidden_state_outputs` — it lives on the
gpu_model_runner instance (gpu_model_runner.py:623/678), unset at plan
time, so the check silently fails and the reserve stays 13,600
(patch 04b). Proven reserve math at 65,536: 13,600 + 4,096 +
5×4096×2 ≈ 58,656 B/token → ~4.15 GiB reserve, arena ~10.3 GiB.
**Verify cheaply before booting:** smoke-test the pure KV logic on CPU
torch with the image's real spec classes — stub `vllm.logger`/`vllm.utils.*`,
register the spec classes in `KVCacheSpecRegistry` with a dummy manager, add
`from __future__ import annotations`, and run synthetic GLM+draft+hidden
spec sets through the patched grouping/layout/builder functions. A 7-minute
boot per hypothesis is the alternative.
**Window discipline:** stop + mask `frank-archon` FIRST and gate on
MemAvailable ≥ 230 GiB before booting the probe (the expert-offload planner
reads MemAvailable; beside-prod boots die at plan time with `ExpertPlanError:
host holds 5113 of 12096 swappable experts`). Save `podman logs glm-probe`
to a file BEFORE teardown — a `podman logs | grep | head` pipeline died once
under pipefail via SIGPIPE before log-save, losing the capture. The abort
path must restore prod (unmask + daemon-reload + reset-failed + start),
never bare-exit after stop+mask. Launcher knobs: `DFLASH_PATCH=1` (mounts
the patched model.py), `DRAFT_MOUNT=1`, `SPECJSON_FILE=…` ({"method":
"dflash","model":"/draft","num_speculative_tokens":8}) — note the launcher's
echo prints `patch=$PATCH_MOUNT` (the COMPILE knob), not DFLASH_PATCH.
Boot timeline: EAGLE3 gate fires ~4 min in at worker init; drafter
torch.compile is the frontier (~380 s).
**Open telemetry discrepancy:** the contaminated window read 1.5–2.1 tok/s
with acceptance 1.00–1.25; Frank live-measured ~18 tok/s decode / ~370 tok/s
prefill. Reconcile with a clean single-request measurement before trusting
either.
Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the same
hardware — its conclusion was drawn from throughput arithmetic and is superseded
by this page's kernel-time measurement),
[[dankserver-frontier-moe-feasibility]] (the earlier fit analysis),
[[VLLM Index]], [[LLM/index|LLM fleet]].
## The deployed shape
| term | value |
| --- | --- |
| payload | `Intel/GLM-5.3-Flash-W4A16-AutoRound` — AutoRound **symmetric int4, group 128**, packing `auto_round:auto_gptq`, 34 shards + 2 extra safetensors, 113,074 tensors, ~170 GiB |
| runtime | `docker.io/tcclaviger/vllm@sha256:da4029666…` (same digest-pinned image as the qwen cutover) |
| engine | TP=2, `--enable-expert-offload --expert-precise-memory`, max-model-len 131,072 x 2 seqs, bf16 KV, util 0.98, NBATCH 2048, prefix caching |
| MoE kernel | `R4dW4a16MoEMethod` → libr4d grouped W4A16 GEMM, 6.234 MiB per expert-rank slot |
| expert cache | 14.51 GiB = **2,383 slots** per rank; 12,096 host-backed experts, **73.64 GiB pinned per rank** |
| unit | `scripts/frank-archon.service` (root unit, `LimitMEMLOCK=infinity` for the pin) |
| measured | **decode 9.9-10.0 tok/s**, **prefill 918 tok/s** (warm, 2.6k-token prompt), KV cache 138,561 tokens |
```mermaid
graph LR
A[Intel AutoRound W4A16<br/>sym int4 g128, auto_gptq] -->|quant_method auto-round| B[inc.py override to inc]
B --> C[inc_wna16_scheme<br/>_resolve_gptq_moe]
C -->|sym int4 g128 + gfx1201 + libr4d| D[R4dW4a16MoEMethod<br/>grouped W4A16 GEMM]
D --> E[ExpertLRUCache<br/>2,383 VRAM slots / rank]
D --> F[73.64 GiB pinned host rows / rank]
E --> G[decode step]
F -->|host misses<br/>1.05% of device time| G
```
## Why this payload and not another
The W4A16 fast path on this fork is reachable **only** through the INC
(AutoRound) quant method. `compressed_tensors` checkpoints of the same model —
for example `dealignai/GLM-5.3-Flash-CYBERSECURITY-W4A16`, 182 GiB — resolve
their MoE through the WNA16 oracle to `TritonWNA16Experts`, which **has no
offload recipe**, and the engine refuses to serve at all: 170 GiB of experts
cannot fit 64 GiB of VRAM, so offload is mandatory, not optional.
The chain that makes Intel's checkpoint work, with the exact evidence:
```
[expert_plan.py] quantization=inc ← inc.py:364 overrides "auto-round"
[inc_wna16_scheme.py:249] INC GPTQ MoE on ROCm: R4dW4a16MoEExperts (libr4d grouped W4A16 GEMM)
[r4d_w4a16_moe.py:682] Using MoEPrepareAndFinalizeNoDPEPModular / R4dW4a16MoEExperts
[expert.py:450] Expert offload: 42 MoE layers staged; 2,383 cache slots in 14.51 GiB planned per rank
[expert.py:413] 42 layers split: 0 VRAM-only experts, 2,383 swap slots, arena 14.51 GiB,
host 73.64 GiB pinned, host rows prepacked on the GPU
```
`R4dW4a16MoEExperts` is what the offload machinery wraps: the offload plan is
format-agnostic but requires the experts class to expose the hot/cold split.
## The numbers, and the wall
Measured on the deployed engine, 2026-09-30 (`bench_glm.py`, real prose, greedy):
| depth | prefill | decode |
| --- | --- | --- |
| 2,621 tok prompt, run 1 (cold cache) | 259 tok/s | 9.88 |
| 2,621 tok prompt, runs 2-3 (warm) | 918 / 885 tok/s | 9.91 / 9.41 |
| 493 tok prompt (telemetry window) | 124 tok/s | 8.90 |
Decode is flat across context and across every arena change — the signature of a
per-step cost that is not KV- or residency-driven at the margin.
### The measured profile, and the real wall
**This section replaces an earlier explanation, which the measurement falsified.**
The first reading of this page bounded the step with throughput arithmetic: 42 MoE
layers x 8 experts x 6.234 MiB = 2.045 GiB per step per rank, 0.82 GiB of it
missed at a measured 0.53-0.66 hit rate, giving "0.82 GiB in ~100 ms = 8.2 GB/s =
the gather is the wall". That arithmetic was circular — it *assumed* the gather
consumed the whole step — and it is wrong. The engine, asked directly, disagrees:
A torch-profiler window (`--profiler-config`, `POST /start_profile` /
`stop_profile`, 496 decode tokens at the deployed shape, rank0 trace parsed for
device-activity time) gives the true split. Total device kernel time **46.18 s /
496 tokens = ~93 ms per token** against a measured ~104 ms step (9.6 tok/s with
the profiler attached):
| ms | % | launches | kernel |
| --- | --- | --- | --- |
| 16,074 | **34.8** | 708,500 | `vectorized_elementwise_kernel` (22.7 us each) |
| 4,602 | 10.0 | 42,000 | `r4d_gemm_moe_w4a16_nt_b16_m64` — the MoE GEMM |
| 3,108 | 6.7 | 174,708 | `elementwise_kernel_manual_unroll` |
| 2,943 | 6.4 | 138,252 | `wvSplitK_hf_sml_` — split-K GEMM (projections / head) |
| 2,829 | 6.1 | 83,160 | router: `sbtopk::gatherTopK` + `warpMergeSortTopK` + `bitonicSortKVInPlace` (1,208 + 1,051 + 559 ms; the 1,051 row is not in the list below) |
| 3,015 | 6.5 | 133,920 | `r4d_mhc_pre_finish/pre_partial/post` — hyper-connections |
| 2,782 | 6.0 | 127,916 | inductor `triton_poi_*` / `triton_per_*` (the fusions that DO happen) |
| **487** | **1.05** | 20,664 | **`r4d_moe_lru_gather_kernel`** — the accused |
| 250 | 0.54 | 20,832 | `_moe_lru_tele_kernel` — the telemetry that measured it (TELEM was on) |
So the gather is **1.05 %** of device time, and `r4d_moe_lru_fused_kernel` (the
LRU *manager*, not the copy) is a further 1,599 ms / 3.5 %. Residency work and the
PCIe link are not the story: 44 % of the budget is **pointwise glue** — bf16<->fp32
conversions, adds, `pow`/`rsqrt` from fp32 RMSNorm, `launch_clamp_scale` quant
passes, `reduce_kernel` means — spread over **~1,428 elementwise launches per
token** (3,734 kernel launches per token in total). The GPU is busy 86 % of the
wall clock, so this is real device work, not launch-gap overhead.
Two consequences that kill three whole families of planned work:
- **Spec decode cannot help this shape, and the MTP measurement was not an
accident.** Pointwise cost scales with the token count, so a draft batch that
verifies 4 tokens pays ~4x the glue for the same per-token output — which is
exactly why MTP4 measured "acceptance 3.76-4.90, decode unchanged". Same reason
the gather was never the bound.
- **Arena size and hit rate are not levers.** Confirmed twice: `--expert-precise-memory`
(arena 13.08 -> 14.51 GiB) moved decode by nothing, and the gather is 1 %.
## Fork work: the compile changelist (WIP, does not boot yet)
The fork compiles its own flagship trunk — `vllm/models/qwen4_exp/amd/model.py`
carries `@support_torch_compile` — and **not** its GLM trunk. The engine says so on
every boot:
```
Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
provides neither a compiled submodule nor breakable CUDA graphs.
Overriding to FULL_DECODE_ONLY.
```
Inductor fusion is therefore the only lever that touches the 44 % pointwise mass.
Three patches are staged in `scripts/patches/glm53-compile-0*.patch`, each carrying
its own evidence header, base/patched sha256 and the next blocker:
| patch | target | fixes |
| --- | --- | --- |
| `-01-model-decorator` | `models/glm5next/common/model.py` | decorates `Glm5NextModel` (env `R4D_GLM_COMPILE`, default on) |
| `-02-trace-safe-probes` | `model_executor/kernels/r4d_lib.py` | freezes 10 capability probes that do `os.path.exists`/`.so` lookups **inside the traced forward** |
| `-03-fullgraph-optout` | `compilation/wrapper.py` | makes `fullgraph=True` opt-out (env `VLLM_COMPILE_FULLGRAPH=0`) |
What the boots established, in order (each one ~7 min):
1. **01 alone dies** on the first cold probe call inside the traced region:
`Dynamo does not know how to trace the builtin 'posix.stat'` —
`r4d_mhc_supported -> r4d_lib.r4d_mhc_enabled -> available -> os.path.exists`.
**`functools.lru_cache` does not protect these probes**: dynamo re-traces the
wrapped body to build its guard even when the cache is warm (it warns about
exactly this), verified in-image — a *warm* lru_cache-calling function still
fails `fullgraph=True`, while a plain module-global read compiles.
2. **01+02 get past it** (`libr4d capability probes frozen: 10 warm [...]` in all
four processes) and die on
`RuntimeError: Graph breaks are not supported with aot compile`, because this
build runs AOT compile by default and AOT refuses breaks.
3. **01+02+03 with `VLLM_COMPILE_FULLGRAPH=0 VLLM_USE_AOT_COMPILE=0` traces the
trunk successfully** (`Dynamo bytecode transform time: 1.08 s`) and then dies on
the decode forward with `KeyError: 'in_proj_qkvbfg_a'` (`Glm5NextModel.forward`
line 748 calling the decoder layer at line 456). `in_proj_qkvbfg_a` is the KDA
layer's merged column-parallel projection (`kda.py:217`, called at :407) — a real
submodule attribute name, also present in the `load_weights` shard table
(`model.py:780-785`). Reading the full traceback (the reported line is inside
dynamo-rewritten code) traces the throw to dynamo's own name→module map:
`torch/_dynamo/output_graph.py` registers into `nn_modules` at :1259/:1384,
looks up with `self.nn_modules[node.target]` at :3000 (a bare-name `KeyError`
when absent), and documents stripping `_buffers[..]/_parameters[..]/_modules[..]`
from those names at :1225. So the compiler is failing to **resolve a submodule by
name** while tracing the GLM decoder layer; whether the root cause is the fork's
module tree or dynamo's registration is not yet settled. The next step is the one
torch itself suggests in that error — `TORCHDYNAMO_VERBOSE=1` for the internal
stack trace (and note the launcher resets `ENVARGS`, so it must be added inside
`serve_glm_autoround.sh`'s COMPILE_PATCH branch rather than in the outer
environment).
4. **04 removes the fused-norm call from the traced region** (`R4D_GLM_FUSE_QKV_RMSNORM=0`:
the GLM MLA block stops calling the fork's fused-norm JIT kernel, whose kernel
object in `models/common/ops/fused_qk_rmsnorm.py` is bound *after* the function
that uses it — `_FUSED_Q_KV_RMSNORM_KERNEL` at line 223 vs its use at line 211).
**It removes a real break** (under `fullgraph=True` the fused call was the first
break dynamo reported; after this patch the report moves on to the workspace
acquisition) — but it is not by itself what blocks a boot, because with breaks
genuinely tolerated the blocker is the name-map `KeyError`. See the run history
below.
5. **A launcher bug that made four boots misleading — worth knowing because the
failure kept moving.** The scratch launcher's compile branches were restructured
on 2026-09-30 and the `-e VLLM_COMPILE_FULLGRAPH=0` line was dropped, so four
consecutive boots mounted the wrapper patch while `_FULLGRAPH` still defaulted to
True. With `fullgraph=True` **every** break is an error, so those boots reported
whichever break came first:
| config | outcome |
| --- | --- |
| `fullgraph=False`, AOT off | breaks tolerated → dies later on the dynamo name-map `KeyError: 'in_proj_qkvbfg_a'` |
| `fullgraph=True`, AOT off | dies on the first break (cell → workspace → compiler-disable) |
So the reported breaks are real *as breaks*, but they are not what blocks a
break-tolerant compile — and an earlier version of this page blamed
`TORCH_LOGS=graph_breaks` for the difference. **That was wrong**; the flag only
logs. The launcher sets the env var again.
**How to see the breaks (and its trap).** `TORCH_LOGS=graph_breaks` names each break
with a user stack, and `VLLM_USE_AOT_COMPILE=0` is required because AOT compile
hard-refuses breaks — but a fullgraph=True run reports the first break as fatal, so
check which configuration you are actually in (`podman exec <ctr> printenv
VLLM_COMPILE_FULLGRAPH`) before concluding a break blocks the boot. For the name-map
KeyError there is a ready-made probe: put `/home/dank/dynamo-probe` on `PYTHONPATH`
with `VLLM_DYNAMO_DUMP_ATTRS=1` (a `sitecustomize.py` — a `.pth` on `PYTHONPATH` is
NOT executed, and the image's own `sitecustomize` is chained so its apport hook
survives) and it prints, at the miss, the node target, the near-miss keys, and
whether another live `OutputGraph` holds the key.
**State of the changelist (2026-09-30, latest).** The trunk **traces**
(`Dynamo bytecode transform time: 1.08 s`) and the boot dies on the dynamo name-map
`KeyError: 'in_proj_qkvbfg_a'`. Established about it, in order:
- It is raised **inside the traced `Glm5NextDecoderLayer.forward`** — the frame chain
is `profile_run → _dummy_run → execute_model → glm4_1v.py:2358 → the compile wrapper
→ eval_frame.compile_wrapper → Glm5NextModel.forward:748 (layer(...)) → module
_call_impl → Glm5NextDecoderLayer.forward:456 → KeyError`. `TORCHDYNAMO_VERBOSE=1`
adds **no** internal stack trace, i.e. it is a plain Python KeyError in the traced
frame, raised in a callee dynamo **inlined** (the reported line is the `def`, which
is how inlined frames surface).
- It is **not** raised from any of dynamo's four `nn_modules` lookup sites:
`/home/dank/dynamo-probe` (a `sitecustomize.py` on `PYTHONPATH`,
`VLLM_DYNAMO_DUMP_ATTRS=1`) wraps three of them and stayed silent across two runs
with the probe confirmed installed in both the API-server and worker processes.
- It is **not** a string-keyed lookup of a literal: `in_proj_qkvbfg_a` appears in the
whole tree only as the KDA layer's attribute (`kda.py:217`, used at `:407`) and in
the `load_weights` shard table (`model.py:780-785`). The four
`proj_name = prefix.split(".")[-1]` sites in the quantization utils do compute
exactly this bare key, but every one of them is guarded (`if proj_name not in
fused_mapping: return`), so they cannot raise it. The pluggable-layer registry is
keyed by class name, not module path.
- It reproduces on **every** run of this configuration (5 of 5), and it is
independent of patches 04/05: with breaks genuinely tolerated, those remove breaks
that were never fatal in the first place — the KeyError is the only thing standing
between this changelist and a boot.
**Next instrument, precisely:** the inlined caller has to be named. Two cheap ways:
`TORCH_LOGS=+dynamo` on a boot and grep the log for the key (dynamo prints inlined
frames with their source), or wrap the inlining candidates in that forward — the mHC
CustomOps (`hc_pre` / `hc_post` / `hc_fused_post_pre`, `layers/mhc.py`) and the
indexer op — with a probe that reports on `KeyError`. Until that names the caller, this
is a dynamo-side name-resolution failure and not a fixable fork bug.
Two verified facts bound the rest of the shape: `suppress_errors` does **not** cover
graph breaks (only `fullgraph=False` does), and the GLM trunk can never satisfy
`fullgraph` because its indexer metadata syncs to host
(`glm5next/amd/sparse_indexer.py:263,419`: `int(...max().item())`) — both are dead
fallbacks on the decode path that the fork author already replaced, so they are not
the blockers they appear to be at first glance.
### Two structural constraints on any further fork work
- **No new HIP kernels are possible.** The image ships `r4d.so` as a binary with no
sources. The obvious second lever — the 9 % router — is gated
`num_experts in {128, 256, 512}` in `router/r4d_route_router.py` and the binary
contains only `r4d_moe_route_kernel<128|256|512>` instantiations. GLM has **288**
experts, so it falls back to the generic sort-based top-k and **cannot be moved to
the fused router without rebuilding libr4d**. Only Python/config/inductor changes
(and runtime-JIT Triton) are available.
- **`--kv-cache-dtype fp8` is a dead end for capacity too.** The plan's KV reserve
is dtype-blind (1.76 GiB at 131,072 either way), so fp8 buys no arena and no
window; it only removes the only MLA decode path gfx12 has (trap 1 above).
### DFlash2 for GLM — what the drafter actually is
`incoai/GLM-5.3-Flash-DFlash2` is **not** a GLM-shaped drafter and needs no MLA/DSA
port: its `config.json` declares `architectures: ["DFlash2DraftModel"]` and
`model_type: "qwen3"` — 5 dense layers, hidden 4096, 32 heads / 8 kv heads,
head_dim 128, sliding attention (window 2048), `is_causal: false`,
`use_cache: false`, `dflash_config` = block 8, conv group 16, selector rank 256 /
top-k 16, and **`target_layer_ids: [5, 14, 24, 33, 42]` against
`num_target_layers: 45`**.
The fork already maps that architecture:
`registry.py:636 "DFlash2DraftModel": ("qwen3_dflash2", "DFlash2Qwen3ForCausalLM")`,
and the spec method is `"dflash"` (`config/speculative.py:67`). So the port is
**target-side only**: the GLM trunk must expose auxiliary hidden states at those five
layer indices, the same interface `deepseek_v2.py` (also MLA+MoE) implements —
`set_aux_hidden_state_layers()` / `get_eagle3_default_aux_hidden_state_layers()` plus
capture at `idx in self.aux_hidden_state_layers` in the layer loop. `glm5next`
implements neither.
**The load was tried, and the gap is exactly one interface.** Payload fetched to
`/home/dank/models/GLM-5.3-Flash-DFlash2` and hash-verified (`model.safetensors`
2,342,169,800 B, sha256 `b038e1d9…` = the Hub's LFS oid). Booting the engine with
`--speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":8}'`
accepts the config and the drafter and dies at worker init with:
```
RuntimeError: Model does not support EAGLE3 interface
```
So nothing about the drafter, the method name, or the payload is wrong — the GLM
trunk simply does not implement the aux-hidden-state contract the dflash runner
requires. The reference is in this same image, twice: `deepseek_v2.py`
(`set_aux_hidden_state_layers` / `get_eagle3_default_aux_hidden_state_layers` at :1948-1951,
capture at :1554 and :1579) and the fork's own `qwen4_exp` trunk. The contract is
small:
- `self.aux_hidden_state_layers: tuple[int, ...]` on the trunk, default `()`, set
by the wrapper's `set_aux_hidden_state_layers()`;
- in the layer loop, `if idx in self.aux_hidden_state_layers: aux_hidden_states.append(<single-stream state entering layer idx>)`
(DeepSeek appends `hidden_states + residual`, with a TP all-gather fixup when the
rank holds fewer tokens than `positions`);
- `Glm5NextModel.forward` returns `(hidden_states, aux_hidden_states)` when the
list is non-empty;
- `get_eagle3_default_aux_hidden_state_layers()` supplies a default triple when the drafter
config omits `target_layer_ids` (this drafter provides `[5, 14, 24, 33, 42]`, so
the runner derives trunk indices `[6, 15, 25, 34, 43]`).
**The GLM-specific unknown** is what "the single-stream state entering layer k"
means for a trunk whose layers carry mHC multi-stream state: DeepSeek's
`(hidden_states, residual)` pair has no counterpart here — GLM's decoder layer
returns `(hidden_states, residual, post, comb)` with `hc_expand`/`hc_pre` mixing
`hc_mult` streams, so the plausible answer is the pre-mHC single-stream input `x`
(4096-wide, matching the drafter's `hidden_size`), not a post-layer value. That is
a semantic choice to validate by acceptance rate, not by inspection.
**Budget no speedup.** Per the profile above, a verification pass pays pointwise
cost proportional to its token count — the same reason MTP4 measured flat — and
MTP4's acceptance was already excellent (3.76-4.90). The port is worth doing as
*enabling work* (it is the prerequisite for any spec decode on this endpoint, and
it is small), not as a performance fix, and the honest expectation on this shape is
flat decode with a prefill regression like MTP's until the pointwise mass is fused.
**The port exists now: `scripts/patches/glm53-dflash-01-aux-hidden-states.patch`** (WIP,
independent of the compile changelist — it patches the same file as compile-01, so
compile-01 goes first if both are applied; base `008eeaa2…` → patched `bc9913a1…`). It
implements exactly the contract read out of this image: `EagleModelMixin` on
`Glm5NextModel`, capture of the single-stream `hidden_states` ENTERING each requested
layer in the loop, the conditional `(hidden_states, aux_hidden_states)` tuple return, and
the three `SupportsEagle3` member attributes plus both interface methods on the
wrappers (capture shape from `deepseek_v2.py:1551-1585`; tuple consumption at
`v1/worker/gpu/cudagraph_utils.py:621`).
**The patch went through five versions, and the decisive discovery was where the
instance actually lives.** The traceback's `set_eagle3_aux_hidden_state_layers(self.model)`
is called on the model the runner LOADED — and for GLM-5.3-Flash that is
`Glm5NextForConditionalGeneration` (the multimodal wrapper; the APIServer log names it),
which holds the text model as `self.language_model = Glm5NextForCausalLM`
(`glm5next/common/model.py:1113`). Patching only `Glm5NextForCausalLM` (v4) still failed
with the identical error because the `isinstance` ran against the wrapper. **v5 fixes
both wrappers** and is the first version to clear the gate (both TP workers log
`Using Eagle3 auxiliary layers from config: (6, 15, 25, 34, …)`).
The full evolution, each an instructive trap:
1. **v1** hand-wrote the two methods the traceback names → identical error. The methods
are not the whole contract.
2. **v2** inherited `SupportsEagle3` as a base (the image's own idiom — `laguna.py:717`,
`mimo_v2.py:834`, …) → dies at class creation with `Cannot create a consistent method
resolution order (MRO) for bases …`. GLM's wrappers already carry four/five
Protocol-derived bases, and simulation confirmed the ForCausalLM's five-base ordering
cannot take `SupportsEagle3` in *any* position.
3. **v3** declared the members as class attributes (MRO-safe) — but only on
`Glm5NextForCausalLM` → identical error (the wrong wrapper, see above).
4. **v4** added all three members (`supports_eagle3`, `has_own_lm_head`,
`has_own_embed_tokens` — a runtime-checkable Protocol's `isinstance` walks ALL
`__protocol_attrs__` *including base-Protocol members* via `getattr_static`, verified
experimentally in-image with stock typing) on `Glm5NextForCausalLM` → identical error.
5. **v5** adds the three attributes to BOTH wrappers + the two methods on both
(`Glm5NextForCausalLM` unwraps to `self.model` — the `EagleModelMixin` trunk;
`Glm5NextForConditionalGeneration` delegates to `self.language_model`, the
`pixtral.py:498` pattern for exactly this wrapper shape) → **gate passes**.
**The lesson generalizes twice over**: when a runtime error names an interface, read the
interface's declaration first — and establish WHICH INSTANCE the check runs against
before patching anything: multimodal wrappers (`…ForConditionalGeneration`) are loaded
models, not pass-throughs, and the text model behind `self.language_model` is invisible
to a check run on the wrapper.
**Retired diagnostic (do not resurrect as-is): the `/probe` startup hook.** An
`op_probe.py` + `sitecustomize.py` pair was mounted to wrap vLLM module forwards and
torch's name-map lookups. It produced no evidence and caused a **fork bomb** (see the
hazard note under experiment-window discipline). Use `TORCH_LOGS` for dynamo questions
instead — it needs no startup hook.
Note also that the KV-cap patch's predecessor arithmetic in
[[dankserver-qwen38-fp8-expert-offload-tuning]] should be read the same way: its
"gather-bound" conclusion came from the same throughput arithmetic that this page's
profiler window has now falsified.
### The KV-cache grouping journey: clearing the unifier, then the kpool tail
With patch 01 v5 the gate passes and the boot moves to KV-cache layout
construction — where the next two blockers live, both in
`v1/core/kv_cache_utils.py`:
1. **The page-size unifier refuses GLM's MLA family.** With a draft model the
spec dict gains non-MLA drafter attention specs and `HiddenStateCacheSpec`
entries, so `_get_kv_cache_groups_glm5_next` (which requires ALL attention
specs to be `MLAAttentionSpec`) bails to the generic grouping path — whose
`unify_kv_cache_spec_page_size` raises
`NotImplementedError: Layer language_model.model.layers.3.self_attn.indexer.k_cache: page size is not divisible… cannot be padded. Padding is only supported for non-MLA attention layers.`
MLA layers cannot be *padded* (sparse MLA indexes whole token rows) — but
they CAN grow `block_size` when the max page is an exact multiple. The
indexer's page is just `tokens_per_state x` the MLA page, so
**`glm53-dflash-02-kv-unify-mla-multiple.patch`** bumps the unified max page
to a multiple of every `MLAAttentionSpec` page before the loop: all MLA-family
layers then take the legal divisible block-growth branch, drafter + mamba
layers pad. The 2026-09-30 window proved it: unify passes, memory profiling
passes, torch.compile passes, and the boot reaches cudagraph capture — the
NotImplementedError is gone.
2. **The kpool tail kernel pins its block.** Cudagraph capture then dies in
`kpool_ops.kpool_decode_update_and_maybe_write_cache_batched`:
`assert tail_kv_cache.shape[2] == pool_size`. The tail cache must view as
`[num_blocks, 2, index_kpool, head_dim]` (`KpoolTailSpec(block_size=index_kpool)`,
`Glm5NextTailCache` in `glm5next/common/attention.py:159`), but the tail's
page (8192 B) *divided* the raised max page, so the unifier's divisible branch
grew its `block_size` by the ratio and the view became
`[num_blocks, 2, index_kpool x ratio, head_dim]`. Verified not the hybrid
promotion: the log carries no "Hybrid KV cache manager is disabled" warning,
so `disable_hybrid_kv_cache_manager` is unset and
`unify_hybrid_kv_cache_specs` never ran. **`glm53-dflash-03-kv-unify-tailpin.patch`**
(= 02 + a `KpoolTailSpec` branch that pads `page_size_padded` instead of ever
growing its block) **verified 2026-09-30**: the assert is gone and the boot
captures the DFlash2 drafter's FULL cuda graphs on both TP workers
(`Capturing model for DFlash2 speculator...`). Supersedes 02 in the repo.
3. **The capacity wall: the generic path cannot alias the mamba states.**
Past capture, the boot dies at `_check_enough_kv_cache_memory`:
`Available KV cache memory: 0.92 GiB` against `31.76 GiB needed at max len
65536` (32.91 GiB at 131,072 — nearly flat, a fixed intercept, not
per-token KV). Two coupled causes, read out of the image:
- **The glm5 tensor layout aliases the mamba states INTO the MLA pages**
(`get_kv_cache_config_from_groups` places every mamba layer at the MLA
offsets — that is why GLM-5.3 fits 32 GiB cards at all, and why the
expert plan's KV registry is calibrated at 13,600 B/token). On the
generic path the mamba groups claim their OWN blocks, so ~30 GiB of
unaliased state pages appears.
- **The plan's KV reserve (`expert_activation_registry.py:_glm53_kv_bytes_per_token`,
a flat 13,600 B/token) knows nothing about the drafter.** The engine caps
KV at the plan's reserve, so even a correct generic-path estimate would
be starved.
Learned en route: the engine AUTO-SETS the attention block size to
**2,432 tokens** (`interface.py:918: Setting attention block size to 2432
tokens to ensure that attention page size is >= mamba page size`), so every
attention group's block counts scale from that floor — a 16-token drafter
block would need 4x the shared `num_blocks`; with the global 2,432 all
groups align.
**`glm53-dflash-04-kv-groups-draft.patch`** (+ **`-04b-kv-registry-dflash.patch`**
for the registry) keeps the aliased glm5 pool and extends it: the grouping
partitions the hidden-state specs (`HiddenStateCacheSpec` is an
`MLAAttentionSpec` subclass — excluded before the exact-type
`type(spec) is MLAAttentionSpec` partition) and the drafter attention
specs; the drafter layers form their own uniform group (per-group pages,
NO unification — the MLA block size stays at the engine-set 2,432 and the
kpool guard `block_size % 512 == 0` is never stressed); the layout returns
a 10-tuple with `draft_group`/`hidden_groups`; the builder sizes ONE
shared backing allocation (`allocate_kv_cache` asserts all tensor sizes
equal) as aliased pool + appended draft/hidden tensors with their own
strides; the memory estimator adds the extras. Patch 04b extends the
registry: 13,600 + 4,096 (drafter 8 kv heads x 128 x K+V x bf16) +
5 x hidden_size x 2 B (the aux hidden-state cache) per token when
`speculative_config.use_aux_hidden_state_outputs` is set.
The pure logic is smoke-tested locally (CPU torch + the image's real spec
classes against synthetic GLM+draft+hidden spec sets): grouping shape,
layout recognition, builder block math and memory accounting all pass.
Real-fit arithmetic: per-token ≈ 58,656 B → reserve ≈ 4.15 GiB at max len
65,536 (arena shrinks ~13.5 → ~10.3 GiB, ~1,650 slots), num_blocks ≈ 29 x
2,432 tokens ≈ 70,500 ≥ 65,536.
All of these mount over site-packages via the scratch launcher's
`KVUNIFY_PATCH=1` knob (host `/home/dank/dflash-patch2/new/…`, patched shas
`5521e50b…` for `kv_cache_utils.py` and `1a623d5c…` for
`expert_activation_registry.py`; apply-verified against the image's pristine
files). The window script's decision pipeline was also fixed for a SIGPIPE
death (`head -80` on a `podman logs` grep pipeline killed one run before
log-save and restore; the log is now captured to a file FIRST, then grepped).
## Measured arms (all 2026-09-30, one boot each)
| arm | decode | prefill | verdict |
| --- | --- | --- | --- |
| baseline int4, MAXLEN 131,072, util 0.92, NBATCH 2048 | 10.03 / 10.08 / 9.41 | 877 / 878 / 885 | qualified incumbent |
| + `--expert-precise-memory` (arena 13.08 → 14.51 GiB) | 9.88 / 9.91 | 918 | **neutral** — proves residency is no longer the binding term |
| + MTP4 spec decode (draft loads, acceptance excellent) | 9.64-11.16 | **255** | **rejected** — see below |
| + MTP4 + explicit `--kv-cache-memory-bytes` (patched cap) | boots, 151,767-token KV | — | enabler for the MTP arm only |
### Why MTP is rejected here (it is not that it fails)
MTP **works** on GLM in this fork: the head loads, the architecture resolves to
`Glm5NextMTPModel`, and acceptance is excellent —
```
SpecDecoding metrics: Mean acceptance length: 3.83-4.90,
Per-position acceptance rate: 0.917/0.833/0.583/0.500 … 0.962/0.846/0.692/0.615,
Avg Draft acceptance rate: 69-97%
```
— and it buys **nothing**: decode 9.6-11.2 tok/s against 10.0 without it, while
prefill **collapses 918 → 255 tok/s**. The reason is structural to offload: a
verification forward gathers the **union** of the batch's experts, so spec
decode amortizes compute but not a residency-bound gather. It also costs 2.6 GiB
of arena. On a compute-bound (fully-resident) shape this would invert; here it
does not.
## Served surface (verified 2026-09-30 against the live engine)
| capability | state | evidence |
| --- | --- | --- |
| `/v1/models` | `glm53-flash`, max_model_len 131,072 | model list probe |
| completion | yes | `bench_glm.py` runs above |
| **tool calls** | **yes** — needs `--enable-auto-tool-choice --tool-call-parser glm47` | a `get_weather` prompt returns `tool_calls: [{"id": "chatcmpl-tool-…", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Seattle\"}"}}]`, `finish_reason: tool_calls` |
| **vision** | **yes** | a 64x64 solid-red PNG is answered "The image appears to be a solid red color…" |
| reasoning channel | **yes** — needs `--reasoning-parser glm47` | without it the thinking leaks into `content` (measured: content began with the answer, then "Let me verify: … = 391", then a bare `</think`) with an empty reasoning field; with it, a train-time prompt answers in `content` ("**Step 1: Add the hours.** 14:20 + 3 hours = 17:20 …") and the chain of thought lands in **`reasoning`** — the vLLM field name the fleet documents (llama.cpp endpoints use `reasoning_content`) |
Without the tool flags the engine answers **HTTP 400** — `"auto" tool choice
requires --enable-auto-tool-choice and --tool-call-parser to be set` — which
breaks every tool-using agent on the endpoint. Both parsers are `glm47`, the
newest GLM dialect the fork ships (`glm45`/`glm47` exist for both).
## Fork traps found (for the engine authors)
1. **`--kv-cache-dtype fp8` is illegal on gfx12 with this model — and it buys
nothing even if fixed.** The `ROCM_AITER_MLA_SPARSE` backend's
`_use_rocm_sparse_triton()` requires a non-fp8 KV cache, and that rope-free
Triton path is the **only** MLA decode available on gfx12: aiter's MLA decode
op is registered under `@if_aiter_supported`, which is gated on **CDNA > 2**
(`is_aiter_found_and_supported()` → `get_cdna_version() > 2`). With fp8 KV
the MTP draft step demands an op that cannot exist here:
`RuntimeError: Worker failed with error ''_OpNamespace' 'vllm' object has no
attribute 'rocm_aiter_mla_decode_fwd''`. KV dtype therefore stays auto/bf16.
(`VLLM_ROCM_USE_AITER_MLA=1` does **not** help — the umbrella gate is the
CDNA check, not the env var.)
- **The capacity payoff is measured at zero.** The point of fp8 KV is fitting
more tokens in the same reserve, and this fork's reserve model is
**dtype-blind**: the fp8 boot printed the *identical* plan line as bf16 —
`KV reserve 1.76 GiB (13600 B/token x 131072 + mamba pages)`. The reserve is
dominated by the per-sequence KDA state (34 linear-attention layers) and the
DSA indexer's pools, not by the MLA KV bytes those 13,600 B/token barely
cover — so halving the KV dtype cannot move it.
- **What a fix would take, if it were ever wanted**: a new fp8 path in the
rope-free Triton kernels (`rocm_sparse_attn_prefill` / `rocm_sparse_attn_decode`
in `v1/attention/ops/rocm_aiter_mla_sparse.py`) that dequantizes pages
in-kernel, plus loosening `_use_rocm_sparse_triton()`. The fork's
`_r4d_fp8_paged_mqa_logits` shows the author already has an fp8 MQA-logits
path, but the sparse-MLA backend is not wired to it. Verdict 2026-09-30:
**do not spend effort here** — the payoff is capacity, and the reserve is
dtype-blind.
2. **The plan's KV reserve under-models this model, and the cap enforces the
mistake.** The plan charges 13,600 B/token + a small mamba term, but GLM
carries a per-sequence KDA state (34 linear-attention layers) and, under MTP,
the draft's KV. `gpu_worker` then *caps* KV at that reserve, so a legal
`--max-model-len` becomes unbootable:
`ValueError: To serve at least one request with the model's max seq len
(131072), (2.07 GiB KV cache is needed, which is larger than the available KV
cache memory (1.74 GiB))`. Raising `max_model_len` only helps past ~440k
tokens (the cap's slope is ~13,845 B/token vs the need's ~12,700 + 0.52 GiB
fixed) — where the reserve would eat a third of the arena.
`scripts/patches/gpu_worker-kv-cap.patch` lets an explicit operator KV
request raise the cap; **it is not part of the deployed shape**, which boots
on the plan's own reserve.
3. **A window is forced, not chosen.** The production `max_model_len 131072` is
what the reserve model requires for this payload (below ~92k the boot
refuses); it is not a capacity decision.
4. **Expert prefetch remains poison** — unchanged from the qwen campaign
(36% precision, +40-51% decode from deleting it). No GLM arm re-opened it.
## Operations
```bash
# rollback to the pre-cutover qwen FP8 + MTP4 shape. The trio is in git; the
# qwen FP8 payload is NOT on disk any more, so re-fetch it first. The R9V state
# dirs are still on the host (/home/dank/r9v-state/):
git show 2339fae -- scripts/ | git apply # or: git checkout 2339fae -- scripts/
./scripts/deploy.sh dankserver --service frank-archon
# payload inventory (34 shards + 2 extras, weight-map index) — in the GLM era
# this was scripts/download-frank-primary.sh (REPO=Intel/GLM-5.3-Flash-W4A16-AutoRound,
# EXPECTED_SHARDS=34, git show f2bcf84). The same-named script today fetches
# Qwen/Qwen3.8-Flash-Next-FP8 (scripts/download-frank-archon.sh:21):
bash scripts/download-frank-archon.sh --verify-only
# live telemetry (needs the profiler window; with_stack=false is mandatory —
# with stack the stop path hangs and the trace buffer grows to tens of GB):
podman logs frank-archon-glm 2>&1 | grep "expert_lru.py"
```
**Experiment-window discipline (learned the hard way, 2026-09-30).** This engine
**ignores SIGTERM**: after `systemctl stop` its container sits in `podman ps` state
`Stopping` for 20+ minutes *while still answering `/health` with 200*. That makes a
liveness probe a lying oracle for a restore — twice a restore script printed
`PROD HEALTHY` while `systemctl is-active` said `failed` and the runtime mask was
still in place. The restore oracle must be a **conjunction**: `is-active` ==
`active` **and** `is-enabled` == `enabled` **and** no `/run/systemd/system/<unit>`
mask symlink **and** the container is owned by the unit and not `Stopping`. The
repair recipe, all four steps needed:
```bash
sudo mv /run/systemd/system/frank-archon.service /tmp/stale-mask-N.service # runtime mask SURVIVES stop and blocks start
sudo systemctl daemon-reload
sudo systemctl reset-failed frank-archon.service # a failed unit refuses to start
podman rm -f frank-archon-glm # SIGKILL; `podman stop` hangs on the same SIGTERM
sudo systemctl start frank-archon.service # then ~7 min to health 200
```
Also: **never let two experiment jobs overlap.** A later job's `podman rm -f
glm-probe` cleanup ran while an earlier job was still mid-flight and deleted its
container, so one whole round produced no evidence at all (`no container with name
glm-probe found`) — check `proc://` for running jobs before opening a window.
**Never put a diagnostic hook on `PYTHONPATH` without a marker guard — it is a fork
bomb.** On 2026-09-30 a `sitecustomize.py` mounted at `/probe` (to wrap dynamo's
name-map lookups and vLLM module forwards) produced a self-replicating chain of
`python3` processes 19+ levels deep inside the probe container: `sitecustomize` is
imported by *every* interpreter, the hook imported torch/vLLM at startup, something in
that import chain spawned a helper python, and the child inherited `PYTHONPATH` (hence
the hook) and spawned again. It was still alive 18 minutes later with the container
still showing `Up`. Symptoms: `ps -eo pid,ppid,comm --forest` shows a deep ladder of
identical interpreters, each child's PPID being the previous one. Guard (now in the
hook): arm once per environment via a marker in `os.environ`, which children inherit —
```python
if os.environ.get("VLLM_PROBE_ARMED") == "1":
pass # an ancestor already armed it; stay inert
else:
os.environ["VLLM_PROBE_ARMED"] = "1"
<install probes>
```
Kill recipe: `podman rm -f <experiment-container>` (the container cgroup owns every
generation), then verify with a precise count over `/proc/<pid>/environ` for the marker
variable — **never** pattern-kill an interpreter name, because the media stack on this
host (romm, jellyfin, plex, glances) runs python in production.
```bash
# boot: ~7 min warm page cache, longer cold; health returns 503 while loading
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health
# experiment windows (production stopped + `systemctl mask --runtime`, which
# self-clears at reboot) ran from the host-side scratch launcher
# /home/dank/serve_glm_autoround.sh: env knobs MAXLEN SEQS NBATCH UTIL KVBYTES
# KVDTYPE SPEC SPECJSON_FILE TOOLPARSER REASONINGPARSER TELEM GCHUNKS GLANES
# PROFDIR GLM_PATCH_MOUNT COMPILE_PATCH GATHER_PATCH DRAFT_MOUNT ENVARGS.
# DRAFT_MOUNT=1 mounts /home/dank/models/GLM-5.3-Flash-DFlash2 at /draft;
# SPECJSON_FILE passes a whole --speculative-config JSON (JSON with no spaces,
# word-split by the launcher's unquoted $EXTRA).
bash /home/dank/serve_glm_autoround.sh
```
Boot gate: the engine refuses a legal-looking shape in two measured ways — the
KV cap above, and the profile-run activation peak (`TritonMLA` indexer scores:
64 heads x 2,048 topk x NBATCH tokens x 2 B = 2.06 GiB at NBATCH 8192), which is
what killed the first two boots at util 0.95.
`PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` is part of the qualified
shape, not a tuning: the plan fills VRAM to its modelled headroom by design, so
the default allocator's fragmentation fails the profile/graph-capture phase
(measured on the deployed unit: "Tried to allocate 662.00 MiB. GPU 0 … 240.00
MiB is free").
The `deploy.sh` e2e gate also drives the admin panel's open-links spec, which
walks every guarded vhost — including `http://192.168.0.112:8989/` on
FrankBook-Pro. When that Mac mini is asleep the spec fails on a `page.goto`
timeout and the deploy reports `DEPLOY INCOMPLETE` even though the engine is
healthy; wake the host (magic packet to `de:6b:6d:c1:52:23`) or re-run
`tests/e2e/run.sh` once it is up. Seen 2026-09-30: 32 passed, this one failed,
and the WoL packet did not wake it (macOS "Wake for network access" needs
toggling on the mini itself).
## Rollback targets
| state | where |
| --- | --- |
| qwen FP8 + MTP4 (2026-09-29 → 09-30) | commit `2339fae`; payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` was deleted from the host (2026-10-18: `/home/dank/models/` has `StillDeadcode`, `rad`, `tpl.jinja`) — rollback = re-fetch |
| R9V radiance Qwen3.8-Flash-Next Q8_0 | [[frank-archon-r9v-radiance-qwen38-q8]], state dirs `/home/dank/r9v-state/{q8xl,uncensored}` |
serverAppConfigs-docs/LLM/frank-archon-llamacpp-qwen38-flash-next-q8.md
-270
View File
@@ -1,270 +0,0 @@
---
type: configuration
title: frank-archon llama.cpp Qwen3.8-Flash-Next Q8_0 Runbook
description: RETIRED 2026-09-26 → 09-27 frank-archon runbook — unsloth llama.cpp (the b11030-mix prebuilt, ROCm gfx120X, bundled runtime, root systemd unit) serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 (175.3 GiB across six split parts) on Dankserver's two Radeon AI PRO R9700s (gfx1201), with the shared Q8_0 MTP drafter (depth 2) chained behind ngram-mod, BF16 vision, and the routed experts + 51B-param n-gram embedding table computed in host RAM. Port 7770, one stream, full 262,144 ctx. Cutover 2026-09-26, replacing the same-day Lucebox DS4.1 deployment; superseded 2026-09-27, and frank-archon runs [[frank-archon-radiance-qwen38-iq4r]] today (2026-10-18).
tags:
- frank-archon
- dankserver
- llamacpp
- unsloth
- qwen38-flash-next
- q8_0
- mtp
- ngram-mod
- spec-decode
- vision
- rocm
- gfx1201
- r9700
timestamp: 2026-09-26
---
# frank-archon llama.cpp Qwen3.8-Flash-Next Q8_0 Runbook
The dankserver LLM endpoint (:7770, alias **frank-archon**) as it stood from
2026-09-26 to 2026-09-27 (retired; the port today serves
[[frank-archon-radiance-qwen38-iq4r]]):
unsloth llama.cpp serving `unsloth/Qwen3.8-Flash-Next-GGUF` at **Q_8**
(the 6-part `Q8_0/` main, 175.30 GiB) on **both** Radeon AI PRO R9700s,
with the **shared Q_8 MTP drafter** (`draft-mtp`, depth 2) layered with
**`ngram-mod`**, and **vision** through `mmproj-BF16.gguf`. It replaces the
Lucebox DS4.1 quadlet that held the port for hours earlier the same day
([[frank-archon-lucebox-ds4]]), which itself replaced
[[frank-medium-radiance-mxfp4]].
## Why this shape
- **One runtime for the fleet.** All three endpoints are llama.cpp now —
one metric namespace (`llamacpp:*`), one reasoning contract
(`reasoning_content`), one tool contract (`--jinja`). The two-day
experiment with a single-vendor engine (lucebox) ended because Qwen
Flash-Next at Q8 on llama.cpp delivers better quality than the DS4.1
fp2/fp3 stream, plus vision, plus the full 262k window, at comparable
speed.
- **The placement law, not a preference:** the routed experts and the
51B-param n-gram (PLE) embedding table are **computed in host RAM**
(`-ot exps=CPU,per_layer_token_embd.weight=CPU`) and never streamed over PCIe. The
x4-pair measurement on the same model shape: streaming loses (3.7 t/s),
computing in RAM does not care (20+ t/s). The dense weights, attention,
projector and MTP head sit on the two cards (~29 GiB); the ~100 GiB of
expert pages live in mmap page cache — `--no-mmap` is forbidden (it
forces the table into anonymous memory; the cgroup kill is silent) and
pinning the PLE table with a GPU target cost 23 → 13.4 t/s when tried.
The tensor names are verified, not guessed: the GGUF v3 header dump
(`dump-gguf.py`, run on the fetched parts 2026-09-26) shows the PLE
table is one tensor — `per_layer_token_embd.weight`, dims
`[160, 320001536]`, Q8_0, 54.4 GiB, exactly the whole of part 3 — and
the model is `qwen4exp` with 512 experts top-10
(`qwen4exp.ple.ngram_size=3`, `heads_per_ngram=8`). The same tensor is
the MTP head's borrow target, so the override is required even if the
placement were optional.
- **Two cards, layer split** (`-ts 1,1`): both R9700s are on CPU root
ports with full PCIe atomics (the ProArt B850-Creator swap,
2026-09-20), but this deployment uses llama.cpp's layer split —
`-ot` tensor overrides do not compose with tensor-parallel mode, and
`exps=CPU` is the override the placement law needs. The 27B-dense
equivalent on this host measured 1991 t/s prefill / 18.8 t/s decode in
this shape; Flash-Next Q8 with CPU experts lands in the same class
(measured numbers below).
- **Spec chain: `draft-mtp,ngram-mod`.** MTP is a property of the FILE:
the shared-Q8_0 head (`mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf`,
2,786,568,256 B) is loaded via `--model-draft` and borrows the target's
embedding tensors — which is why `per_layer_token_embd.weight` must be
CPU-resident in the override set (a GPU-resident borrow crashes the
draft load with `invalid vector subscript`). `ngram-mod` is draftless,
so it chains legally behind the drafter. Depth 2: each longer
verification round re-batches the CPU expert reads, so depth buys less
than bandwidth here. **The drafter must sit on ROCm1** — the head also
borrows the target's `output.weight`, which a `-ts 1,1` split places on
the last-layers card; with tensor overrides set the draft scheduler runs
`n_copies==1`, and a draft context elsewhere aborts at load
(`ggml-backend.cpp:941`, "pre-allocated tensor (output.weight) in a
buffer (ROCm1) that cannot run the operation"). The 2026-09-26 cutover
crash-looped 4× on `ROCm0` before the probe confirmed `ROCm1` loads —
the frank-medium precedent (drafter on the last-layers card) was right.
- **Vision, explicitly.** This build never autoloads projectors — without
`--mmproj` the server comes up healthy and text-only and answers image
requests with 500 (the frank-medium 2026-09-17 trap). The BF16 projector
(907,542,944 B) is named in the serve script.
## Deployment
|Piece|Value|
|---|---|
|Unit|`scripts/frank-archon.service` — root systemd unit, `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity`, relinked to `/etc/systemd/system/`|
|Serve script|`scripts/serve-frank-archon.sh` — validates binary + all 6 parts + drafter + mmproj before exec|
|Env tunables|`scripts/frank-archon.env` — paths and numbers only; bind is `BIND_HOST` (never a bare `HOST`: HIP reads it)|
|Runtime|`/home/dank/GitPackages/unsloth-llama-b11030/llama-server` — unsloth b11030-mix prebuilt (commit 6ba30d05b), ROCm gfx120X, bundled runtime via `LD_LIBRARY_PATH`|
|Payload|`/home/dank/models/Qwen3.8-Flash-Next-GGUF/` — `Q8_0/` (6 parts), `MTP/mtp-...-shared-Q8_0.gguf`, `mmproj-BF16.gguf`; fetched by `scripts/download-frank-archon.sh` (sha256 + byte-count verified)|
|Port / alias|7770 / `frank-archon` — unchanged from the lucebox and radiance eras; no client moves|
|Context|262,144 × 1 slot (`PARALLEL_SLOTS=1`, one stream, the model's full native window)|
|Sampling|Greedy server default (deterministic e2e); Qwen-recommended sampling rides `models.yml` `compat.extraBody` per request|
|Template|The GGUF's embedded template via `--jinja`; reasoning → `reasoning_content` (`--reasoning-format deepseek`)|
|Observability|`frank-archon` Prometheus job at `192.168.0.235:7770` — cross-host LAN scrape (the netavark published-port quirk that forces `host.containers.internal` is same-host only); exports `llamacpp:*`|
### Cutover mechanics
The deployer (`deploy_llm_archon` in `scripts/deploy-remote.sh`) gates on
every artifact at its declared byte count (`stat -c %s`, never `wc -c` —
which read the whole 377 GiB lucebox file for minutes on 2026-09-26),
retires the lucebox user quadlet to dated home trash
(`~/.AGENTTRASH/20260926-llamacpp-cutover/`), waits for port 7770 to be
truly free, relinks + enables + restarts the root unit, and waits up to
1800 s for `/health` (cold start page-ins ~175 GiB and builds the graphs).
### Preflight the GGUF without downloading it
GGUF metadata lives in the header and HF honours ranged GETs, so a few MB
answers every sizing question before a 13 GB transfer
(`Range: bytes=0-8000000`). Parse: magic `GGUF`, u32 version, u64
n_tensors, u64 n_kv, then key/value pairs (string = u64 len + bytes; ARRAY
= u32 elem type + u64 count; BOOL is 1 byte — the type hand-written parsers
forget). Walk arrays by byte size even for a summary: a naive parser
desynchronises on `tokenizer.ggml.tokens` and lands gigabytes off. Read:
`general.architecture`, `<arch>.block_count`, `<arch>.context_length`,
`<arch>.nextn_predict_layers` (MTP is a property of the FILE — the `-mtp`
build carries `blk.<n>.nextn.*`; `--spec-type draft-mtp` then loads the
head against the MAIN model's weights, and `--model-draft` would load the
file twice), `<arch>.head_count_kv`, `<arch>.key_length`/`value_length`,
`<arch>.full_attention_interval`, `<arch>.ssm.state_size`/`group_count`/
`time_step_rank`, and for the projector `clip.projector_type` +
`clip.has_vision_encoder`.
KV arithmetic for the hybrid (16 × (3 Gated DeltaNet + 1 Gated Attention)):
only full-attention layers take KV
(`attn_layers × n_head_kv × head_dim × 2 × bytes_per_elem`; f16 = 2 B,
q8_0 = 1.0625 B, q4_0 = 0.5625 B); DeltaNet state is a fixed per-slot
buffer (~0.16 GiB/sequence at 48 layers/128 d_state/48 v-heads f32); the
MTP draft context costs ≈ 4 KiB/token. `--parallel N` DIVIDES `--ctx-size`
between slots — it does not multiply.
## Verification (2026-09-26 deploy)
- `/v1/models` reports `frank-archon`.
- Chat completion returns `content` + `reasoning_content`.
- Tool call returns well-formed `tool_calls` (`--jinja`).
- Real image request answered from the projector (not a health check —
health cannot see vision).
- Spec chain live: `draft acceptance` lines in the unit journal; ngram-mod
has no startup banner (identify behaviourally: `draft_n` > 0 where text
repeats).
- `reasoning_effort: high` → 200 (the template's `{xhigh, medium, low}`
gate is folded by `reasoningEffortMap` in `models.yml`).
- VRAM/RAM headroom: `rocm-smi` per card + `free -g` after the first long
completion.
_**Measured (2026-09-26 battery, 6/6 pass).** Load: ~7 s warm (mmap from
page cache; the fetch's own verify pass pre-warms it — a cold first load
pages in 175 GiB). Decode: **11.8 t/s** `predicted_per_second` at
depth-0 prose, 128-token generation, draft_n 109 / accepted 72 (0.66 on
reasoning-heavy prose; **0.864 mean len 2.73** on the tool-call task).
Vision: the probe image (green square centered on a red field) described
exactly. VRAM: GPU0 29% / GPU1 36% of 32 GiB (~9.3 / ~11.5 GiB — the
output-head card carries the drafter, as designed; ~22 GiB free per
card). RAM: ~11 GiB used, ~226 GiB page cache (the payload itself),
available ~235 of 246 GiB. `reasoning_effort: high` → 200. Two cutover
traps for the record: a LAST-ACK tail from a LAN client held
`wait_port_free` until it cleared (a foreign process on the port gates
the deploy correctly — but the deployer's `/health` wait cannot tell WHO
answers; after any aborted deploy, confirm the responder is the unit's
own MainPID), and llama-server's SIGTERM cleanup hung after
spec-decode service (logged `cleaning up before exit` and never exited —
`kill -9` was needed; `systemctl stop` relies on TimeoutStopSec)._
**KV law (from the -lv 4 load log, cross-checked by the GGUF header):**
29,696 B/token total ctx-scaled VRAM — 24,576 B main KV (only 12 of 48
layers carry attention, `full_attention_interval=4`; 2 KV heads x 256,
f16) + 3,072 B indexer + 2,048 B draft KV. Everything else (weights
~6.7 GiB, recurrent state 0.33 GiB, compute buffers ~4.1 GiB) is
ctx-independent. At 262,144: 7.25 GiB scaled, 18.75 GiB total, 9.37
GiB/card. At 100,000: 2.77 GiB scaled, 14.3 GiB total, 7.1 GiB/card.
`--parallel N` splits the same pool — slots do not multiply KV.
**Reshape (later the same day): four full slots, V q8_0, hot expert
blocks on-GPU.** `CONTEXT_SIZE=1048576` / `PARALLEL_SLOTS=4` — four
conversations, each the FULL native 262,144 window (per-slot window
unchanged, so no client or models.yml implication; slots divide the
pool, never multiply it). `CACHE_TYPE_V=q8_0` with K held at f16: main
KV drops 24,576 → 18,816 B/token (−23.4%), pool law 23,936 B/token.
The shared MTP drafter's KV ignores the V type (measured: draft pool
2,048 MiB = the f16 law at 1,048,576 cells — one layer's worth, the
miss costs ~0.5 GiB on ROCm1). At 4×262,144 the pool is ~23 GiB across
both cards: main 9,408 MiB + indexer 1,536 MiB per card, draft
2,048 MiB on ROCm1.
Expert blocks stay GPU-resident — and the measurement below shows it
cannot matter *which* ones. `-ot` granularity is the whole
per-block expert tensor (512 experts share one tensor), so "critical
experts" = blocks co-located on the card that holds their layers:
blocks 0-4 on ROCm0, 45-47 on ROCm1 (measured 2.42/2.49 GiB per
block). The placement law that makes it work: b11030 wants ONE
comma-joined `--override-tensor` (repeats log DEPRECATED), and the
loader applies the FIRST matching pattern per tensor — the
block-specific patterns MUST precede the broad `exps=CPU` sweep, or
the sweep silently swallows them (verified live: broad-first left
every expert on CPU with no warning anywhere in the log).
Measured headroom, deployed fill **5/3** (blocks 0-4 on ROCm0, 45-47
on ROCm1 — the operator directed the fill past the 3-4 GB checkpoint
floor the same day, "shift a few more experts"): card0 2.53 GiB free
at boot, 2.23 GiB under load with the vision graph exercised; card1
3.31 GiB at boot, 3.15 GiB under load. Card1 cannot take another
block — 3.31 − 2.49 = 0.82 GiB, and the vision graph (~0.43 GiB) plus
boot drift alone exceed that — so 5/3 is the physical limit at the
4×262144 / V-q8_0 shape. Earlier 4/3 state for reference: card0 5.02
GiB boot free. Live numbers (4/3 era): content decode 13.05 t/s
(acceptance 0.58, mean len 2.16), reasoning phase 7.24 t/s
(acceptance 0.74, mean len 2.48), two concurrent full generations both
delivered, prompt cache reusing 42/64 tokens across requests. All
three Prometheus fleet targets up; the 5/3 deploy re-verified chat,
vision and the target on the same battery class.
**Which blocks, measured (same day):** asked whether the `-ot` regex
could target the *most active* blocks, I instrumented the live server
instead of guessing — `scripts/frank-archon-expert-activity.py`
samples llama-server with AMD IBS (`perf record -e ibs_op// --data`)
while 4 mixed concurrent generations run, maps each sampled load
address through `/proc/<pid>/maps` back to its backing tensor (extents
derived from consecutive GGUF header offsets — never from a quant
type-size table, whose errors silently mis-bucket sample tails), and
aggregates per block and per expert. Verdict, 75 s window / 2.7M
expert-tensor samples: **per-block read volume is FLAT, 0.93×–1.08× of
mean across all 40 CPU-resident blocks** (0-4 and 45-47 absent —
GPU-resident). MoE top-k routing fires every block on every token, so
"most active layers" do not exist at `-ot` granularity: the regex can
express any 8-block set, and any set is equivalent to any other — the
lever is COUNT, already at the 5/3 VRAM ceiling. The real skew is
per-EXPERT (top 10% of the 48×512 (block,expert) cells carry 40.5% of
samples; hottest cell 0.1%), below the reach of a tensor-level
override. Not a benchmark: it instruments production serving in place
at negligible IBS overhead; re-measure any future checkpoint or
traffic mix with `frank-archon-expert-activity.py <pid>` (or
`--analyze-only` against the saved sample file in `/tmp`).
## Rollback
The lucebox DS4.1 image `localhost/lucebox-rocm:gfx1201` is still in dank's
rootless storage; the 378 GiB payload is not (Dankserver's `/home/dank/models/`
holds only `StillDeadcode`, `rad` and `tpl.jinja` as of 2026-10-18), and the
quadlet-era repo files were deleted by `8500600` rather than trashed — there is
no `dankserver/.AGENTTRASH/20260926-llamacpp-cutover/` (see
[[frank-archon-lucebox-ds4]]). Restore = recover the `.container` from
`git show 8500600^`, stop the root unit (`systemctl stop frank-archon`; the
fleet law forbids `systemctl disable` on these hosts — it removes the unit
entry, which here is the symlink into the checkout), `daemon-reload`, start the
user quadlet. The Qwen payload (~179 GiB) is no longer on disk either, so a
forward re-swap is a re-fetch plus a redeploy.
## Fleet context
- [[frank-dragoon-unsloth-qwen38-27b-q8xl]] — the era's Dankclaw :7770, the same
b11030-mix binary family on gfx1151 (retired 2026-09-28; frank-dragoon runs
[[frank-dragoon-flash-server-v2-native]] today).
- [[frank-zealot-gemma4-12b]] — the era's Dankclaw :7771, mainline llama.cpp on
the 3090, and then the fleet's only audio-capable endpoint. Both are history:
frank-zealot serves [[frank-zealot-llama-prism-bonsai2-pq20]] today and no
fleet endpoint takes audio (the audio path left with Gemma4 on 2026-10-01).
- The harness contract lives in `/home/blank/.omp/agent/models.yml`. The era's
`llamacpp (dankserver)` numbers (`contextWindow 262144`, `maxTokens 65536`,
`input: [text, image]`) are long gone — that provider reads
`contextWindow: 200000`, `maxTokens: 16384`, `input: [text, image]` as of
2026-10-18.
serverAppConfigs-docs/LLM/frank-archon-lucebox-ds4.md
-222
View File
@@ -1,222 +0,0 @@
---
type: configuration
title: frank-archon Lucebox DeepSeek V4 Runbook
description: RETIRED 2026-09-26 frank-archon runbook (active for hours on 2026-09-26) — the Lucebox runtime (github.com/Luce-Org/lucebox, local image localhost/lucebox-rocm:gfx1201) serving DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, 43-layer MoE, 256 routed experts, top-6) on Dankserver's R9700 with the routed experts computed on CPU from the mmapped page cache. Replaced the same day by [[frank-archon-llamacpp-qwen38-flash-next-q8]] (llama.cpp Qwen3.8-Flash-Next Q8_0); the image is still in rootless storage but the payload is gone from the host, and frank-archon runs [[frank-archon-radiance-qwen38-iq4r]] today (2026-10-18).
tags:
- frank-archon
- dankserver
- lucebox
- deepseek-v4
- ds4
- rocmfp23
- gfx1201
- r9700
- quadlet
- moe
timestamp: 2026-09-26
---
# frank-archon Lucebox DeepSeek V4 Runbook
> [!warning] RETIRED 2026-09-26 (same day it went live)
> Replaced by [[frank-archon-llamacpp-qwen38-flash-next-q8]] — unsloth
> llama.cpp serving Qwen3.8-Flash-Next Q8_0 on both R9700s. This runtime
> served for hours; the image `localhost/lucebox-rocm:gfx1201` is still in
> dank's rootless storage, but the 378 GiB payload was deleted from the host
> (`ls /home/dank/models/` on Dankserver, 2026-10-18: `StillDeadcode`, `rad`,
> `tpl.jinja` only), and the quadlet-era repo files were deleted by `8500600`
> (2026-09-26) — they survive only in git history, not in a `.AGENTTRASH`
> directory (`dankserver/.AGENTTRASH/` holds only `20260916-vllm-radiance`).
The dankserver LLM endpoint (:7770, alias **frank-archon**) as it stood on
2026-09-26: the
[Lucebox](https://github.com/Luce-Org/lucebox) runtime serving
`Lucebox/DeepSeek-V4.1-Flash-ROCMFP23-GGUF` (377.2 GiB, 405,349,859,328
bytes, LFS oid
`2daea6ca1715bc987aae201e63445d3f7abdc2337c05f5f569f5e50237ce8be2`) on a
single Radeon AI PRO R9700 (gfx1201). It replaced
[[frank-medium-radiance-mxfp4]] (vllm-radiance TP=2), retired earlier the
same day.
## Why this shape
DeepSeek-V4.1-Flash is a 43-layer MoE: 256 routed experts + 1 shared expert
per layer, top-6 routing. The ROCMFP23 quantization is fp2 gate/up with fp3
down experts (Q2_0_ROCMFP2 type 107 + Q3_0_ROCMFPX type 104). At 377.2 GiB
the model fits neither the 63.7 GiB of combined VRAM nor the 246 GiB of RAM,
so it is served the way llama.cpp serves big models by default: the GGUF is
mmapped, pages pull from disk/page cache on demand, dense work (attention,
shared expert, embeddings, output projection) and the MLA KV cache run on
the R9700, and the routed experts are computed on CPU. The 246 GiB RAM
(upgraded for this deployment) holds the working set of active experts; the
rest streams from the 1.8 T NVMe.
Lucebox's `--ds4-fused-decode` requests a monolithic GPU load; on a model
this size that allocation fails and the backend falls back to **hybrid
expert placement + layered decode** — that fallback is the intended serving
mode here, not an error.
## Image
`localhost/lucebox-rocm:gfx1201`, built on the host from
`github.com/Luce-Org/lucebox` `Dockerfile.rocm`:
```
git clone --recurse-submodules https://github.com/Luce-Org/lucebox.git
cd lucebox
# Three local modifications (tracked at the time as
# dankserver/frank-primary/lucebox-gfx1201.patch, deleted by 8500600):
# 1. Dockerfile.rocm drops the ROCm *-dev apt packages (the 7.14.1-full
# base already ships them in /opt/rocm; the apt repo does not serve
# them) and the three runtime math libs.
# 2. Dockerfile.rocm installs the official cmake 3.31.6 binary (jammy's
# 3.22 predates the `block` command ROCm 7.14 hipblaslt-config.cmake
# requires, CMake >= 3.25).
# 3. ggml-cuda.cu: `ggml_cuda_node_launches_nothing` moved above the
# `#ifdef USE_CUDA_GRAPH` guard — `ggml_cuda_node_is_noop` calls it
# unconditionally, but the macro is never defined by the build
# system, so the in-guard definition was dead and the use failed.
git apply ../lucebox-gfx1201.patch
podman build --build-arg LUCE_HIP_ARCHES=gfx1201 \
--build-arg ROCM_VERSION=7.14.1-full \
-t localhost/lucebox-rocm:gfx1201 -f Dockerfile.rocm .
```
- The prebuilt `ghcr.io/luce-org/lucebox-hub:rocm` is **not usable as-is**:
it is built for gfx1151 + ROCm 6.4.1, and gfx1200/gfx1201 are not
code-object compatible.
- `ROCM_VERSION=7.14.1-full` matches the host driver's ROCm 7.x line (the
radiance image's core-7.14 userspace runs on this host). A 6.4.x userspace
on a 7.x driver can segfault at model load. Only `-full` tags exist for
7.14.x.
- The binary lands at `/opt/lucebox-hub/server/build/luce_server`.
## Model
`/home/dank/models/DeepSeek-V4.1-Flash/DeepSeek-V4.1-Flash-ROCMFP23.gguf`,
fetched with `huggingface_hub` (the xet-native path — plain `curl` full GETs
400 against HF's xethub CDN for this repo; see the fetch note in
`scripts/hf-fetch.sh`). Byte count and sha256 are deploy-gated in
`deploy_llm_archon`.
## Deployed flags (2026-09-26)
From `dankserver/frank-primary/frank-primary-entrypoint.sh` (deleted by `8500600`;
recover it with `git show 8500600^:dankserver/frank-primary/frank-primary-entrypoint.sh`) — the unit
bypasses the image's entrypoint, which only forwards a fixed Qwen/DFlash
flag set and cannot express the DS4 flags):
| Setting | Value |
|---|---|
| Model | `DeepSeek-V4.1-Flash-ROCMFP23.gguf` (377.2 GiB) |
| Device | `--target-device hip:0` (`HIP_VISIBLE_DEVICES=0`; the Raphael iGPU at index 2 stays out) |
| Window | `--max-ctx 131072` (the qualified Strix Halo window; DS4's MLA KV is compressed, so the pool is small) |
| Chunk | `--chunk 8192` (the qualified profile's chunk) |
| KV dtype | `--cache-type-k q4_0 --cache-type-v q4_0` (the qualified launch) |
| Decode | `--ds4-fused-decode` (falls back to hybrid expert placement on this host) |
| Prefill | `--ds4-prefill sparse` (the batched prefill; `exact` runs at token-wise decode cost) |
| Routing | `--ds4-expert-top-k 6` (the model default; do NOT reduce on adaptive qtype-105/106 artifacts — measured degenerate output at top-4, DS4.md 2026-08-05) |
| Draft | none — no DSpark draft is published for V4.1-Flash; plain autoregressive decode |
| Identity | `--model-name frank-archon`, `--host 0.0.0.0 --port 7770` |
## API surface
OpenAI Chat Completions (streaming + tool metadata), `/v1/models`,
`/health`, OpenAI Responses, Anthropic Messages, and Open WebUI model
metadata — the fleet contract is unchanged from frank-medium.
## Cutover procedure
1. Image built and model fetched (gates in `deploy_llm_archon`).
2. `deploy.sh` → `deploy_llm_archon`: stops and **masks** the radiance
unit (`frank-medium`), waits for :7770 to free, installs
`dankserver/frank-primary/quadlet/frank-primary.container` (also deleted by
`8500600`; recover it from `git show 8500600^`), starts
`frank-archon.service`, waits for health (1800 s start period — first
start page-ins the dense weights and builds the graphs).
3. Verify: `/health` 200 is not enough — a real chat completion, a tool
call, and an Open WebUI completion.
4. Retire the radiance unit to the home trash (never `rm`); the MXFP6
checkpoint payload did not stay — it is gone from the host (see Rollback).
## Rollback
Unmask and start the radiance unit (`systemctl --user unmask frank-medium &&
systemctl --user start frank-medium`) and stop/mask frank-archon. The
`localhost/radiance-vllm:0.13.0-e060895` image is still in dank's rootless
storage; the `/models/Qwen3.8-27B-PARO-MXFP6` checkpoint is not — Dankserver's
`/home/dank/models/` holds only `StillDeadcode`, `rad` and `tpl.jinja` as of
2026-10-18, so this rollback needs a re-fetch of the payload.
## Verification (2026-09-26)
_Filled in after the cutover: health, real completion, tool call, Open WebUI
completion, measured prefill/decode, `systemctl --user is-enabled/is-active`
state._
## Dual-R9700 follow-up (2026-09-28 probe era)
After this single-card runbook was retired, the DS41 engine got a dual-R9700
attempt on the `feat/ds41-engine` branch (`main` rejects the checkpoint with
`unsupported model architecture 'deepseek41'`). Findings that outlive the
probe:
**Budget both cards explicitly, or you waste half the box.** The engine
sizes the streamed-expert cache LAST, after the model, so uncapped it absorbs
every spare byte on the stream device — symptom: card1 at 33.0/34.2 GB
looking busy while card0 sits at 14.2/34.2 GB; the busy card is holding a
cache of things still streaming from NVMe. Pin all four:
| env | card | meaning |
|---|---|---|
| `LUCE_EXPERT_BUDGET_MB` | hip:0 | primary hot experts (~11.96 MiB each) |
| `LUCE_EXPERT_SECONDARY_BUDGET_MB` | hip:1 | secondary experts (needs `LUCE_DS4_MOE_TP*`) |
| `LUCE_EXPERT_STREAM_CACHE_MB` | hip:1 | cap the auto-sized stream cache |
| `LUCE_EXPERT_STREAM_DEVICE=1` | hip:1 | which card holds the slots and computes them |
Working full-card shape: 21000 + 20000 + 8000 → card0 ~28.6/31.86 GiB,
card1 ~27.3/31.86 GiB, resident experts 3,360 of 15,360.
**Expert TP DOES work on gfx1201+gfx1201.** A retired quadlet recorded
`LUCE_DS4_MOE_TP_INPROC` as "qualified for gfx1151+gfx1201, not
gfx1201+gfx1201" — that is WRONG; peer access negotiates and the secondary
tier comes up clean: `[deepseek4-moe-tp] enabled mode=in-process local=hip:0
secondary=hip:1`. It is the only lever that materially raises resident
expert count, and resident count is the entire bottleneck. Confirm every run
with `expert ownership:` and `hybrid experts ready:` lines.
**The placement file must live under the mounted models dir.** JSON
`{"owners": [...]}`, 15,360 entries indexed `layer*384+expert`, values
`primary`/`secondary`/`stream`. It is a ranking, not a promise: each device
keeps what fits its budget and demotes the rest. Anywhere outside
`/models/...` gives `cannot open expert placement` and a SILENT single-tier
fallback — a silent-wrongness failure, not a crash.
**V4.1 refuses layer split and layer-major prefill** — only `--ds4-prefill
exact` (token-by-token) is legal on a 32 GiB card and `--ds4-fused-decode`
is refused there. With ~22% of experts resident each token pulls ~187 cold
experts × ~12 MiB; if `/proc/<pid>/io` `read_bytes` climbs into terabytes
the model is larger than the box's RAM and you are in page-cache thrash at
the NVMe ceiling — a capacity verdict, not a tuning problem. Low GPU util
with a pegged link is the EXPECTED signature of PCIe-expert-stream-bound
decode, not a broken run.
**Harness traps, each of which produced a wrong number:** never `env -i` the
podman call (rootless podman runs as `dank`; `env -i HOME=/root` makes
PODMAN try `/root/.config` and die `stat /root/.config: permission denied`
before the container exists — pass container env with `-e`); never put a
large prompt on curl's argv (`Argument list too long` at ~128 KB — use
`--data-binary @file`); size prompts by growing the file until `wc -c`
reaches the byte target AND assert depth against the engine's own
`usage.prompt_tokens` (nominal 4 chars/token was really 5.73; a chunk loop
that appends a fixed ~152-byte line per nominal 200-byte chunk undershot
30%); the server runs as the container's PID 1, so tear down with
`podman rm -f`, never host `pkill`; drop page cache between arms
(`sync; echo 3 > /proc/sys/vm/drop_caches`) or the first arm reads warm and
every later one is a lie.
Read the placement log — it is the whole diagnosis
(`arch=deepseek41 layers=40 embd=5120 experts=384 used=6 ff=2304
hash_layers=0` — hash_layers=0 is correct for V4.1; `engram: layers=[1,14]
… 2 table(s) embedded (not mapped)` — the 189.13 GiB Engram tables are never
mapped, so they do not evict the expert page cache).
serverAppConfigs-docs/LLM/frank-archon-r9v-radiance-qwen38-q8.md
-296
View File
@@ -1,296 +0,0 @@
---
title: frank-archon — R9V radiance-vLLM Qwen3.8-Flash-Next Q8_0
service: frank-archon
host: dankserver
port: 7770
status: retired
deployed: 2026-09-27
supersedes: "[[frank-archon-r9v-radiance-qwen38-uncensored]]"
tags: [llm, frank-archon, r9v, vllm, radiance, gfx1201, q8]
---
# frank-archon — R9V radiance-vLLM Q8_0 (dankserver, RETIRED 2026-09-27 → 09-30)
The Dankserver LLM endpoint **was** the **R9V radiance-vLLM fork** (retired
2026-09-30, when [[frank-archon-glm53-flash-w4a16]] took the port; the engine
live today, 2026-10-18, is radiance with
[[frank-archon-radiance-qwen38-iq4r]])
(`drwolfen/radiance-vllm-qwen4exp`, docker-pinned prebuilt image bundle
`2dac17a2`) serving the **unsloth Qwen3.8-Flash-Next-GGUF Q8_0** payload
(175.30 GiB, 6 GGUF shards, sha256-verified per part) with the blockfp8
MTP4 head and the mmproj, on **both R9700s** (gfx1201, BDFs `0000:03:00.0`
/ `0000:07:00.0`) as `frank-primary` on **:7770** (the `frank-archon` wire alias
arrived with the 2026-10-09 rename).
Cut over from the uncensored IQ4_XS payload on 2026-09-27 at Frank's
direction. Rollback target at the time:
[[frank-archon-r9v-radiance-qwen38-uncensored]] — its state dir
(`/home/dank/r9v-state/uncensored/`, with `setup.json` and
`restore-config-20260927.env`) is still on the host as of 2026-10-18, but the
IQ4_XS payload itself is not (`/home/dank/models/` holds only `StillDeadcode`,
`rad`, `tpl.jinja`), so that flip now needs a re-fetch.
## Why the swap
The uncensored IQ4_XS payload measured 59–96 tok/s decode with
draft-acceptance swings; the Q8_0 payload trades VRAM-resident expert
headroom for precision — quantization to Q8_0 doubles expert bytes per
block, so the tiered-expert residency budget halves — the probe-era
11 tok/s decode was residency-bound (expert streaming, not compute).
The "Known costs" section below carries the accepted trade.
## Production shape (repo-tracked)
The repo carries a **thin** wiring; the engine shape lives on-host:
| Piece | File | Law |
|---|---|---|
| Unit | `scripts/frank-archon.service` | Root unit, `User=dank`, `LimitMEMLOCK=infinity`, `ExecStop=podman stop -t 120 frank-archon-r9v`, `Restart=on-failure` |
| Env | `scripts/frank-archon.env` | Production overrides ONLY: bind 0.0.0.0, :7770, alias `frank-archon`, container `frank-archon-r9v`, GPU BDFs, dev-overlay adapter, `R9V_FULL_MUTABLE_CACHE=0`, `R9V_PLE_READY_TIMEOUT=1800`, prefix caching ON, sharp chat template |
| Serve | `scripts/serve-frank-archon.sh` | Layers setup state under repo env → execs the fork's own `scripts/launch.sh` via the docker→podman shim → `docker wait` |
| Deployer | `scripts/deploy-remote.sh` `deploy_llm_archon` | Gates on setup state ready, image digest, PLE bytes, all six shard files + mmproj; MainPID false-green check |
**Why the tuned shape lives on-host:** the q8 profile's setup run
qualified the `R9V_*` config (tiered-expert placement, per-rank offload
budgets, 80-slot tiered cache, PLE **pinned** residency 54.4 GB, MTP4,
parser wiring, `MAX_MODEL_LEN=131072`). It is written to
`/home/dank/r9v-state/q8xl/setup.json` and the serve script layers it
UNDER the repo env (repo keys win). Copying those values into the repo
would drift from the qualification — don't.
## Spec decode: MTP4 only — the ngram-mod ask is NOT expressible
llama.cpp chains drafter types (`--spec-type draft-mtp,ngram-mod` on
frank-zealot/large). The fork's `speculative_config` is a **single dict**
(one method per engine): `mtp`, `ngram`, `ngram_gpu`, `dflash`, … are
mutually exclusive — no chaining seam exists in the fork. What IS active:
- **MTP4** (`R9V_MTP_SPEC_TOKENS=4`, blockfp8 draft head, draft TP=2) —
measured mean acceptance length 3.19–5.00 during qualification.
- **The PLE n-gram** — architectural (`ngram_embedding` module), rides
along automatically; it is not the llama.cpp `ngram-mod` matcher and
cannot be "added" as a second spec method.
- `ngram_gpu` exists as an ALTERNATIVE single method (fork's GPU n-gram
proposer) — swapping MTP4 for it is an either/or, not a chain.
## Sharp chat template (fleet parity with frank-dragoon)
`scripts/qwen-sharp-v22.4.0.jinja` is served via the launcher's opt-in
`R9V_CHAT_TEMPLATE_FILE` passthrough (fork commit `30d4a32`): the file is
mounted read-only at `/opt/r9v/share/chat-template.jinja` and passed as
`--chat-template`, which wins over the tokenizer dir's
`metadata/chat_template.jinja`; `R9V_CHAT_TEMPLATE_KWARGS` carries
`--default-chat-template-kwargs {"preserve_thinking":true}` (the template
defaults `preserve_thinking` true natively; request-level kwargs still win
per request). The template folds reasoning efforts (high/xhigh/max →
xhigh, minimal/low → low), matching `models.yml`'s `reasoningEffortMap`.
The sha-pinned `metadata/chat_template.jinja` in the package is NEVER
edited in place — mutating it drifts the qualification.
## Two q8-only runtime laws
1. **Quantized inject shard** — the Q8_0 payload quantizes BOTH fused
hyper-connection shards (the IQ4_XS package ships inject as F32), so
the shard-uniformity check rejects the fused projection at load.
`R9V_DEV_QWEN4_EXP_ADAPTER_PY` (fork commit `363d606`) mounts a
dev-overlay adapter that dequantizes a quantized inject shard
in-stream; uncensored payloads pass through unchanged.
2. **`R9V_FULL_MUTABLE_CACHE=0`** — the runtime overlay's `env.always`
injects the uncensored 160-slot full-mutable cache geometry, which the
q8 profile's 80-slot tiered cache fails with a baseline mismatch. The
launcher passes this key AFTER the overlay envs (docker last-wins).
## Fork checkout + shim (on-host, outside the repo)
- Checkout: `/home/dank/GitPackages/radiance-vllm-qwen4exp` (fork commits
`363d606` dev-overlay, `2fe6dbb`, `50a79ec` qualification doc,
`30d4a32` chat-template passthrough, `2de0357` PLE-timeout forwarding —
local-only, remote is GitHub)
- State: `/home/dank/r9v-state/q8xl/setup.json` (0600, `ready: true`)
- Model dir: `/home/dank/models/Qwen3.8-Flash-Next-Q8_0-R9V/` — 175.30 GiB
payload as six `Q8_0/Qwen3.8-Flash-Next-Q8_0-0000N-of-00006.gguf` shards
+ mmproj; PLE derived at the pinned size 54,400,261,120 B
- Docker→podman shim: `/home/dank/GitPackages/radiance-vllm-qwen4exp/.podman-shim/docker`
(PATH-prepended by the serve script). The R9V tooling is docker-pinned;
the shim carries three laws:
1. **Image ID law** — the bundle is a containerd/OCI docker-save; docker
keys the image by its OCI MANIFEST digest (`sha256:2dac17a2…`), podman
by config digest but preserves the manifest digest in **RepoDigests**.
The shim answers `image inspect sha256:X` from RepoDigests and rewrites
bare `sha256:X` run refs to `repo@sha256:X`.
2. **LogConfig law** — podman inspect answers
`HostConfig.LogConfig.Config: null` where docker returns the log-opts
map (the doctor crashes on `.get("Config", {}) == null`). The shim
records `run --log-opt` pairs and normalizes inspect output.
3. **docker info law** — podman lacks `SecurityOptions`/`ServerVersion`;
the shim answers rootless + `27.5.1` (liveness probe).
- venv: `~/.r9v-venv` (huggingface_hub for fetch/verify)
## Bring-up battery (qualification, 2026-09-27)
- Setup `ready: true`; all 20 payload artifacts size+sha256 verified
(175.30 GiB total across the six shards).
- Vision: 64×64 red PNG → "Red" (79 tok, 7.4 s end-to-end on the scratch
port). Prose: 100 tok in 9.1 s ≈ 11.0 tok/s wall **on a cold cache with
reasoning tokens included in the wall clock** — decode-rate context, not
a bench (no throughput bench was run, per Frank's direction).
- MTP4: mean acceptance length 3.19–5.00 across the battery.
- PLE pinned proof: `mapping_bytes = mapping_rss_bytes =
54,400,261,120` (fully RSS-resident); doubling for re-pin would need
108,800,522,240 ≤ available 140,876,808,192; registration took 446 s.
## Memory shape (measured)
- Load: 21.84 GiB / worker at ~158 s; cold start → serve ~16 min.
- PLE pinned residency 54.4 GB in host RAM (vs the uncensored payload's
ssd-mmap residency).
- Context envelope: `R9V_MAX_MODEL_LEN=131072`; harness `models.yml`
carries `contextWindow: 131072`, `maxTokens: 36864` (compaction-gap
math: 70% × 131,072 = 91,750 + 36,864 = 128,614 ≤ 131,072). The KV
envelope at 131,072 is smaller than the uncensored 196,608 shape.
- Prefix caching OFF at cutover (`R9V_ENABLE_PREFIX_CACHING=0`), CACHE192
off — superseded 2026-09-28: prefix caching is ON (see bench matrix
below).
## Prefill / decode bench matrix (2026-09-28, scratch port 7773)
Production stopped for the window (fleet bench law); one matrix plus three
follow-up runs, fork's own bench-pp + decode_speed harness, corpus-sliced
prompts with the fork's canonical decode rounds. Baseline E0 = production
shape (chunk 4096, 80-slot sync cache, MTP4):
| Config | Override | PP8K tok/s | PP131K tok/s | Decode | Verdict |
|---|---|---|---|---|---|
| E0 | — | 954.6 | 883.0 | 141.9 ms/step ≈ 23.8 t/s eff | baseline |
| E1R | chunk 8192 | **1201.4 (+25.9%)** | 844.6 (−4.4%) | 158.5 ms (−8.5% t/s) | short-prompt win, decode tax |
| E2 | chunk 16384 | — | — | — | boots; 33.3/34.2 GiB VRAM (0.93 free) → requests hang/OOM |
| E1/E3 | chunk 8192/32768 | — | — | — | PLE registration blew the 600 s engine timeout (see law below) |
| E4 | cache async=1 | — | — | — | doctor: cache-prepare backend requires synchronous single-fill LRU |
| E5/E6 | 96/112 slots | — | — | — | doctor: capacities (48, 64, 80) only; expert-budget [108, 198] rank maxima — **80 is the policy max, prod is at it** |
| E7 | MTP depth 6 | 843.1 | 819.6 | 454–489 ms/step (3.7x worse) | depth 4 stays |
Laws from the matrix:
- **4096 is the prefill-chunk ceiling twice over**: the engine caps
scheduled tokens by the speculative config, and E2's VRAM (0.93 GiB free
at 16384) shows bigger chunks OOM at request time. 883–955 tok/s is what
this payload shape delivers at long context.
- **Unqualified shapes fail mid-request, not at health** — E2 and E7 both
answered /health 200 and then died streaming.
- **PLE registration can exceed the 600 s engine timeout**
(`VLLM_PLE_OFFLOAD_READY_TIMEOUT`) under page-cache churn — measured 740 s
after a bench teardown; a production boot failed on it the same morning
(unit "Deactivated successfully", no error in journal — the serve script
exits 0 on the wait). Fixed with `R9V_PLE_READY_TIMEOUT=1800` in
`frank-archon.env` (forwarded by launch.sh `2de0357`).
Prefix caching (E8/E9, `R9V_ENABLE_PREFIX_CACHING=1`, same port): **zero
VRAM cost** (hits reuse freed KV blocks; load VRAM identical to E0),
cold path unchanged (first request ≈ cold bench), reuse hits large — 32k
prompt TTFT 44.2 s → **3.6–4.3 s** (9x), 131k 439 s → **6.9 s** (63x).
Hit pattern: rounds 1–2 miss, round 3+ hit (identical-prompt and
conversation-growth probes both). For the harness's resend-every-turn
traffic that is a ~40–440 s TTFT win per steady-state turn; the one-turn
miss costs nothing extra. **DEPLOYED 2026-09-28** (Frank's go-ahead): the
repo env sets `R9V_ENABLE_PREFIX_CACHING=1` (commit `41e9189`), verified
on :7770 (`--enable-prefix-caching` in the container argv, health 200,
live completion answering).
## Authoring a new profile (the qualification pipeline)
Order matters at every step:
1. **Package manifest** `packages/models/<pkg>/package.json` — target shards
+ aux artifacts with sha256. Verify every target's byte count AND sha256
after fetch (`curl -fL -C -` per shard; never trust a fetch without `-f`).
2. **Placement bootstrap** from GGUF headers: parse all shards' tensor
tables (a KV-only shard1 is normal for split parts; each part carries its
own tensor table). Per-expert full per layer =
(`ffn_gate_exps` + `ffn_up_exps` + `ffn_down_exps`)/512; rank slices
rank0 = full×416//640, rank1 = full−rank0. Hot counts =
floor(measured_budget / per-expert total). **Anchor budgets to the
MEASURED hot_bytes of the qualified sibling profile, never invented.**
3. **Trim + validate**: `tools/trim_experts.py --hot-counts N,M`, then
`expert_budget.validate_manifest(m)` +
`expert_budget.expert_memory(m, slots, {cache_rank}, runtime=<runtime.json>)`
(runtime/catalog are keyword-only). Declared hot_bytes must match
computed or it raises.
4. **PLE derive** (containerized `tools/prepare_ple.py` with
`--expected-bytes/--expected-type/--expected-shape`); setup skips
re-derivation when the file exists with matching size.
5. **profile.env**: set values BEFORE `source dual-r9700/profile.env`
(set-first-wins; an empty string does NOT survive `:=`).
The canonical bootstrap.json wrapper schema is exactly
`{schema: "r9v.placement.v1", id, model_package, hardware, artifact_path,
artifact_sha256, scope}` with `hardware: "dual-r9700-32gb-asymmetric"`; the
wrapper `id` MUST equal the id in the profile.json placement descriptor
(setup rejects mismatches: "does not match profile placement"). Never
fabricate qualification metadata — use
`scope: "Unranked routing bootstrap only; does not certify fit or headroom"`.
GGUF parsing laws for the bootstrap: expert tensor names are
`blk.0.ffn_up_exps.weight` — split on "." gives FOUR parts, layer number is
parts[1] (a 3-part assumption crashes `int("")`); outer-KV value-type sizes
`{0:1,1:1,2:2,3:2,4:4,5:4,6:4,7:1,8:str,9:arr,10:8,11:8,12:8}`; tensor byte
extents from consecutive OFFSETS, never a type-size table (mis-buckets tails
silently); the PLE Q8_0 tensor on this arch is shape (160, 320001536),
54,400,261,120 B, an entire shard.
Setup/start: `./r9v setup <alias> --state-dir <dir> --model-dir <dir> --image
sha256:<digest> --local-image --reuse-from <sibling-model-dir>
--accept-model-license` (aux artifacts reuse byte-identical files; F16 vs
Q8 mmproj will not reuse). Start requires the SAME `--state-dir`; fixed
placement via `R9V_EXPERT_MANIFEST_PATH` skips headroom/catalog/seed raises.
The host-only doctor FAILs when another instance holds the cards (it wants
3 GiB free per rank; production at ~30 GiB/card is a hard wall) — a second
instance CANNOT coexist, so qualification requires a stop+mask window for
the incumbent, then unmask+restart.
**Qualification gate (unqualified → qualified):** first start runs the
workload battery (PONG / tool_calls / vision / long-ctx), the PLE pinning
proof (`VmLck` ≈ PLE bytes in `/proc/<container-pid>/status` when
RESIDENCY_MODE=pinned), and a VRAM headroom snapshot (floor 3.3 GiB). If
headroom is violated, lower hot counts via re-trim and re-derive
MAX_EFFECTIVE. Only then record state `ready: true` and author the runbook.
Remote-execution gotchas from the same windows: `nohup X &` inside an ssh
command races ssh teardown — the script can die before writing a byte; run
long remote scripts synchronously, or setsid + disown, or via a system
unit. zsh login shells abort on unquoted `===` separators; grep globs need
quoting through ssh+zsh layers.
## Ops
- **Restart**: `sudo systemctl restart frank-archon` (root unit; the serve
script re-gates everything, relaunch through the shim, waits).
- **First start after boot** re-warms ~175 GiB of page cache; expect
minutes to /health; `TimeoutStartSec=1800`.
- **Scratch/probe**: NEVER probe against the prod state dir. Copy it, set
`R9V_HOST_PORT` to a free port and `R9V_CONTAINER_NAME` to a scratch
name, and stop the probe before any deploy — a probe holding :7770
produces a false-green deploy (the 2026-09-26 lesson, guarded by the
MainPID check).
- **Rollback to uncensored**: point
`scripts/frank-archon.env` `R9V_STATE_FILE` back at
`/home/dank/r9v-state/uncensored/setup.json`, drop the two q8-only keys
(adapter, FULL_MUTABLE_CACHE) and the template keys, redeploy — the
uncensored state dir is intact on disk; the IQ4_XS payload is not (it was
deleted in the 2026-10-17 cleanup), so this rollback re-fetches first.
- **Models on the wire**: served alias `frank-primary` (the StarCraft wire
names arrived 2026-10-09, `7c77326`); thinking channel is `reasoning` (vLLM);
tool parser `qwen3_coder`.
## Known costs (accepted at cutover)
- **Residency-halved experts** — Q8_0 doubles expert bytes vs IQ4_XS, so
the tiered cache holds half as many experts in VRAM at the same budget;
the probe's 11 tok/s decode was residency-bound, and the production
number depends on the re-qualified budgets in the q8 state.
- **128K envelope** — down from the uncensored 196,608 (which itself was
down from 262,144 on llama.cpp).
- **Reference quant** — back to the unsloth Q8_0 weights (no
abliteration); answer-quality profile differs from the uncensored
incumbent by design.
- **Docker-pinned fork** — updates flow through image bundles + the shim,
never a bare container run; the launcher stays the single argv owner.
serverAppConfigs-docs/LLM/frank-archon-r9v-radiance-qwen38-uncensored.md
-138
View File
@@ -1,138 +0,0 @@
---
title: frank-archon — R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS
service: frank-archon
host: dankserver
port: 7770
status: retired
retired: 2026-09-27
superseded_by: "[[frank-archon-r9v-radiance-qwen38-q8]]"
supersedes: "[[frank-archon-llamacpp-qwen38-flash-next-q8]]"
tags: [llm, frank-archon, r9v, vllm, radiance, gfx1201, retired]
---
# frank-archon — R9V radiance-vLLM (dankserver, RETIRED)
> **RETIRED 2026-09-27** — superseded the same day by the Q8_0 cutover,
> [[frank-archon-r9v-radiance-qwen38-q8]] (itself retired 2026-09-30; the live
> frank-archon runbook today, 2026-10-18, is
> [[frank-archon-radiance-qwen38-iq4r]]). The uncensored state dir and frozen
> config are still on disk (`/home/dank/r9v-state/uncensored/` — `setup.json`,
> `restore-config-20260927.env`); the IQ4_XS **payload is not** — it was deleted
> in the 2026-10-17 cleanup. The rollback procedure lives in the Q8_0 page's Ops
> section, with a re-fetch as step one.
The Dankserver LLM endpoint **was** the **R9V radiance-vLLM fork**
(`drwolfen/radiance-vllm-qwen4exp`, docker-pinned prebuilt) serving
`Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS` — the abliterated
Qwen3.8-Flash-Next at IQ4_XS (92.39 GiB, 3 GGUF shards) with the blockfp8
MTP4 head and the F16 mmproj — on **both R9700s** (gfx1201, BDFs
`0000:03:00.0` / `0000:07:00.0`) as `frank-primary` on **:7770** (the
`frank-archon` wire alias arrived 2026-10-09).
Replaced llama.cpp b11030 (Qwen3.8-Flash-Next Q_8) on 2026-09-27 at Frank's
direction; downtime window ~1 h 20 min (killed 02:47, healthy 04:11 PDT).
Rollback target: [[frank-archon-llamacpp-qwen38-flash-next-q8]] — the
llama.cpp trio (unit/env/serve) is at commit `4992e5a`+ (the repo-side
wiring; host unit was the relinked one); the GGUF payload is no longer on
disk (Dankserver's `/home/dank/models/` holds only `StillDeadcode`, `rad`,
`tpl.jinja` as of 2026-10-18), so that rollback is a re-fetch plus a redeploy.
## Production shape (repo-tracked)
The repo carries a **thin** wiring; the engine shape lives on-host:
| Piece | File | Law |
|---|---|---|
| Unit | `scripts/frank-archon.service` | Root unit, `User=dank`, `LimitMEMLOCK=infinity`, `ExecStop=podman stop -t 120 frank-archon-r9v`, `Restart=on-failure` |
| Env | `scripts/frank-archon.env` | Production overrides ONLY: bind 0.0.0.0, :7770, alias `frank-primary` (the wire id became `frank-archon` on 2026-10-09), container `frank-archon-r9v`, GPU BDFs |
| Serve | `scripts/serve-frank-archon.sh` | Layers setup state under repo env → execs the fork's own `scripts/launch.sh` via the docker→podman shim → `docker wait` |
| Deployer | `scripts/deploy-remote.sh` `deploy_llm_archon` | Gates on setup state ready, image digest, PLE bytes, shard files; MainPID false-green check |
**Why the tuned shape lives on-host:** the profile's setup run qualified a
104-key `R9V_*` config (tiered-expert placement, per-rank offload budgets
146.25/78.75 GiB, CED on, MTP4, parser wiring, `MAX_MODEL_LEN=196608`). It
is written to `/home/dank/r9v-state/uncensored/setup.json` and the serve
script layers it UNDER the repo env (repo keys win). Copying those values
into the repo would drift from the qualification — don't.
**Self-healing law:** the launcher `--detach`es; the serve script therefore
blocks on `docker wait`. Without the wait, a crashed engine is a green
unit. `docker wait` cannot stop the container child, hence `ExecStop`.
## Fork checkout + shim (on-host, outside the repo)
- Checkout: `/home/dank/GitPackages/radiance-vllm-qwen4exp`
- State: `/home/dank/r9v-state/uncensored/setup.json` (0600, `ready: true`)
- Model dir: `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS/`
(92.39 GiB payload + `r9v-data/per_layer_token_embd.iq4_nl.bin` 26.82 GiB
PLE, sha256 `34fa36f8…`)
- Docker→podman shim: `/home/dank/GitPackages/radiance-vllm-qwen4exp/.podman-shim/docker`
(PATH-prepended by the serve script). The R9V tooling is docker-pinned;
the shim carries three laws:
1. **Image ID law** — the bundle is a containerd/OCI docker-save; docker
keys the image by its OCI MANIFEST digest (`sha256:2dac17a2…`), podman
by config digest but preserves the manifest digest in **RepoDigests**.
The shim answers `image inspect sha256:X` from RepoDigests and rewrites
bare `sha256:X` run refs to `repo@sha256:X`.
2. **LogConfig law** — podman inspect answers
`HostConfig.LogConfig.Config: null` where docker returns the log-opts
map (the doctor crashes on `.get("Config", {}) == null`). The shim
records `run --log-opt` pairs and normalizes inspect output.
3. **docker info law** — podman lacks `SecurityOptions`/`ServerVersion`;
the shim answers rootless + `27.5.1` (liveness probe).
- venv: `~/.r9v-venv` (huggingface_hub for fetch/verify)
## Verified at cutover (2026-09-27)
- Setup: **PASS=28 WARN=3 FAIL=0** — 20/20 artifacts size+sha256 verified
(~93 GiB), PLE derived at exact pinned size, CED projector verified.
- First-start doctor battery: **PASS=34 WARN=8 FAIL=0**, including
`runtime-worker-transport`: **pinned UVA reads, H2D copies, TP
all-reduce verified inside serving workers** on both BDFs.
- Wire: `/v1/models` → `frank-archon`; `/health` + `/metrics` 200
(Prometheus scraping); "Say PONG" (non-thinking) → `'PONG'`, stop;
thinking requests populate `reasoning` (vLLM channel).
- Memlock: the full tiered-expert load ran to healthy **under a plain dank
session** before the unit existed — the pin path does NOT hard-require
`LimitMEMLOCK=infinity` on this host, but the root unit keeps fleet
parity and headroom for the pinned reserve.
- e2e gate: 32 passed / 1 failed — the failure is `11-admin open links`,
which needs FrankBook-Pro (192.168.0.112) awake; the machine was down and
WOL from both hosts failed. Suite minus that spec: **31 passed** (the
one-time 12-crawl flake re-passes in isolation).
## Memory shape (measured)
- Both cards ~29–30 GiB used of 31.86; engine budget
`CPU_OFFLOAD_GB_BY_DEVICE=146.25,78.75` (UVA host pin ≈ 43 GiB on rank 0
+ 16 GiB PLE pinned reserve; PLE residency mode `ssd` mmap+trim).
- `MemAvailable` floor: 56.3 GiB (`R9V_MIN_HOST_AVAILABLE_BYTES`).
- Context envelope: 192,608 native window served as `MAX_MODEL_LEN=196608`;
harness `models.yml` carries `contextWindow: 196608`, `maxTokens: 57344`
(compaction-gap math: 70% × 196,608 = 137,625 + 57,344 ≤ 196,608).
## Ops
- **Restart**: `sudo systemctl restart frank-archon` (root unit; the serve
script re-gates everything, relaunch through the shim, waits).
- **First start after boot** re-warms ~93 GiB of page cache; expect several
minutes to /health; `TimeoutStartSec=1800`.
- **Scratch/probe**: NEVER probe against the prod state dir. Copy it, set
`R9V_HOST_PORT` to a free port and `R9V_CONTAINER_NAME` to a scratch
name, and stop the probe before any deploy — a probe holding :7770
produces a false-green deploy (the 2026-09-26 lesson, now guarded by the
MainPID check).
- **Qualification artifacts** (kept on-host):
`/home/dank/r9v-state/uncensored/qualification-*`,
failure bundles `failure-*` (review before sharing — raw logs).
- **Models on the wire**: served alias `frank-archon`; thinking channel is
`reasoning` (vLLM), tool parser `qwen3_coder` (not qwen3_xml like the
vllm030 wiring had planned).
## Known costs (accepted at selection)
- **Abliterated weights** (Dyluhn uncensored build) — not the reference
model; answer-quality deltas are possible.
- **192K envelope** — down from 262,144 on llama.cpp.
- **Docker-pinned fork** — updates flow through image bundles + the shim,
never a bare container run; the launcher stays the single argv owner.
serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-iq4r.md
-154
View File
@@ -1,154 +0,0 @@
# frank-archon — Radiance (StillDeadcode), publisher Qwen3.8-Flash-Next iq4r, 2× R9700 TP=2
**Status: LIVE — deployed 2026-10-15.** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 2× Radeon AI PRO R9700 (gfx1201), TP=2
The in-house w4dn8 conversion of the Swift1.5 finetune
([[frank-archon-radiance-swift15-w4dn8]]) is replaced by the **engine
author's published container**:
[`StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe`](https://huggingface.co/StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe)
— the official `Qwen/Qwen3.8-Flash-Next` bf16 checkpoint rad-converted by
StillDeadcode. Same port, alias, container name, systemd unit, digest-pinned
engine pin **1.0.13** (`stilldeadcode/radiance@sha256:4c3f484564…`), and
**every serving argument carried verbatim** from the 2026-10-13 shape.
Rollback: re-download the w4dn8 `.rad` from the HF mirror + point
`MODEL_FILE` at it (the local copy was purged 2026-10-17, see Rollback).
## The payload
`qwen3.8-next-flash-fp8-iq4r-moe.rad` — **121,969,901,568 B = 116,322 MiB**,
sha256 `0af5e96244e80c21ac8edca1719dae49b8a201b94a2f3994c3e24026ceaa4d20`
(Hub LFS oid), fetched byte-exact by `scripts/download-frank-archon-iq4r.sh`
(hfget resume + size + sha256; staging dir
`/home/dank/models/StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe/` keeps the
card, LICENSE, and recipe `qwen4exp-w4nl64-i8-hc8m` as provenance).
| Plane | Encoding (publisher recipe `qwen4exp-w4nl64-i8-hc8m`) |
| --- | --- |
| Routed experts | 4-bit codes into a 16-level **non-uniform** codebook after a 128-point Walsh–Hadamard rotation, one E4M3 scale per 64 weights; codes by GPTQ (10M-token calibration, per-expert down-proj Hessians) + 3 coordinate-descent sweeps |
| Protected experts | **ten** per layer, whole-expert bf16 (down-projection energy) |
| Trunk + lm_head | int8 W8A8, scale per 128 columns searched; HC mixtures E4M3 |
| PLE n-gram | fp8 fixed-scale, in-container (served `--ngram-placement ram`, same 47.7 GiB pin law) |
| Speculator | the model's own MTP head (card default depth 3; served at 2) |
| Vision | bf16 vision tower, in-container |
**Base model: the OFFICIAL (censored) Qwen3.8-Flash-Next** — the fleet's
only frontier model at base alignment; the abliterated family lives on
frank-dragoon. Card quality: full-vocab teacher-forced KL vs bf16 over
~75K positions — mean 0.0789 all-tokens / 0.0239 assistant turns, top-1
agreement 92.85 %.
## Why this swap
- **Publisher-graded precision**: the w4nl codebook + coordinate-descent
refinement is the engine author's own quality ceiling for this
checkpoint; the card's KL ladder (p99 1.65, p99.9 5.33) is published and
reproducible, where our w4dn8 gate measured against our own reference.
- **Formal parity**: the served file is the one the engine author supports
— decode/prefill behavior on 1.0.13 is their tested path (the card's run
line speaks the 1.0.13-era flags; the `--vram-weights-mib`/`--vram-kv-mib`
pair we ship is accepted by pin 1.0.13 per the 2026-10-06 A/B, so the
shape needed no translation).
- **Cost**: the Swift1.5 finetune's refusal-bending behavior leaves this
endpoint (it returns to the official base alignment); the 4-slot/200k
economics and all shape laws carry over unchanged.
## Shape (unchanged from 2026-10-13)
All of it, verbatim — the payload moved 116,416 → 116,322 MiB, inside the
same budget:
- `--tp 2 --tp-wire wht6`, `--max-num-seqs 3`, `--max-model-len 200000`
(capacity contract 3 × 200,000), `--kv-cache-dtype fp8`
- `--placement expert_tiered --host-pool-mib 44928` + explicit
`--vram-weights-mib 25600 --vram-kv-mib 5120` (30,720 of the engine's
30,969 MiB ceiling; pool = contract + 1.3 %)
- `--num-speculative-tokens 2`, `--max-num-batched-tokens 2048`
(MNBT 2048 law), prefix cache 8 GiB host + 64 GiB disk,
`--ngram-placement ram`, froggeric v22.5 `--override-chat-template`
- TTM preflight floor 150,000 / target 204,800 MiB; pinned ledger
2 × 44 GiB pool + 47.7 GiB PLE + 8 GiB prefix ≈ 147 GiB (see
[[frank-archon-radiance-swift15-w4dn8]] for the ledger arithmetic and
the KV dtype law — both are engine properties, payload-independent)
The publisher's card run line (`--host-pool-mib 12288`,
`--expert-vs-cache-ratio 0.82`, seqs 8, spec 3, ~32 GB host RAM) is their
shape on stock 32 GB cards; the fleet ships its own measured shape against
the same file.
## Cutover record — 2026-10-15
1. `rm` (Frank's one-time authorization): the raw
`ukisai/Swift1.5-Qwen3.8-Flash-Next` BF16 checkpoint (336 GiB, spent
conversion source) and `qwen4exp-fp8-native.rad` (173 GiB, two eras
back) left `/home/dank/models` — 150 GB free → 658 GB free. The fp8-native
payload is **no longer on disk**: rebuild =
`scripts/download-frank-archon.sh` + `scripts/build-frank-archon-rad.sh`.
2. Fetch: `scripts/download-frank-archon-iq4r.sh --fetch` as transient unit
`iq4r-fetch` (detached-job protocol; log
`/home/dank/logs/iq4r-fetch.log`, rc sentinel `…rc`).
3. Deploy: `scripts/deploy.sh dankserver --service frank-archon` — launched
detached (the swap restarts the very endpoint the operator session runs
on; the harness timeout catches the session on return).
4. **Boot ledger (2026-10-15, first boot, pin 1.0.13 — the w4nl codebook
loaded, no pin bump)**: `kv_attn` fp8 page **26.00 KiB** at **607,740
tokens** (151,935 blocks = 3.77 GiB) — byte-identical to the w4dn8 boot,
the sizing law carries; `kv_qsa_bkey` 3.25 KiB at the same token count;
`kv_gdn_state` 54 MiB × 6 slots, `kv_gdn_conv` 1.76 MiB × 3,
`kv_qsa_tail` 19.5 KiB × 6; pool **5.00 GiB**, prefix checkpoints 8 ×
56 MiB; the sizing line reads `kv cache 5.00 GiB … backs 3 sequences all
at 200000 tokens at once, which is the worst case`; samplers print **9
rows** per rank (3 × (1+2), inside the wht6 25-row bound);
`speculation: 'mtp' drafts 2 token(s) a step in 2 serial pass(es)` +
prompt-lookup fallback; slab **17,610 (rank0) / 17,658 (rank1) slots**
(64 shadow) — above the w4dn8 record of 17,548 (ten bf16-protected
experts per layer replace twenty-four, so more of the 4-bit table
resides), pinned pool 7,532/7,484 slots; KV lending **4.22 GiB**
(3,571/3,568 mover slots); arenas 896.02/988.17 MiB charged before the
pair split; prefix cache RECOVERED 900,176 blocks + 126 checkpoint slots
from the 64 GiB disk tier (same arch + tokenizer as w4dn8 — the tier
survived the payload swap); froggeric template 28,234 B; vision encoder
live (≤ 4,194,304 px, 16,384 patches/pass). Health gate passed in ~100 s
(radiance binds the port after the full load).
5. **Vision probe (2026-10-15)**: 300 px solid-red PNG via `image_url` —
reasoning stream: "a solid fill of a deep, saturated red hue". Content
came back EMPTY at `max_tokens 64`: the thinking budget consumed all 64
(`reasoning_tokens: 64`, finish `length`) — fleet law, probe thinking
models with ≥ a few hundred tokens.
6. **e2e gate**: deploy exited DEPLOY INCOMPLETE — 30/33 Playwright green,
3 red (`12-crawl`, `13-sso` search SSO, `16-marreta`) on DANKCLAW
surfaces (vhosts intermittently returned `ERR_CONNECTION_RESET` from the
workstation); frank-archon's own health gate passed. The swap touched no
dankclaw runtime file (only homepage services.json text). Census of the
Caddy/auth stack pending; see `log.md`.
7. **Post-boot measurements (pending)**: decode ladder (99.0 t/s single
@ 43.5k, 93.3 @ 197k, 3-way 113.9/43.3/10.9 = 18.4 t/s aggregate —
these are the **w4dn8** numbers, the record to beat, not this file's),
slab-limited prefill (22208/3,397 → 25600/3,584 t/s A-cold 43.5k), draft
acceptance, and `scripts/probe-frank-archon-kv.py` leg D.
## Rollback
Revert the swap commit (`scripts/frank-archon.env` MODEL_FILE line) +
redeploy: `/home/dank/models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad`
(116,416 MiB), KLD-gated, same shape — **no longer on disk**. The 2026-10-17
disk purge (Frank's approval, trash-staged then emptied: 763 GiB free after)
removed it; the byte-exact copy lives on the Hub as
`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`
(`qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`)
and re-fetches through `scripts/download-frank-archon-iq4r.sh`'s
hfget/verify pattern retargeted at that repo. The same purge deleted the
`radiance-state` kvcache tiers, `~/.radiance-cache-w4a8-*`,
`~/.cache/frank-primary`, and `/opt/vllm/moremodels` (GLM-4.7 GGUF) — all
derived or retired. The fp8-native era is rebuild-only (see the cutover
record). The w4dn8 runbook keeps every shape law, the KV dtype table, the
slab ladder, and the tuning levers:
[[frank-archon-radiance-swift15-w4dn8]].
## Related
- [[frank-archon-radiance-swift15-w4dn8]] — the replaced payload and the
full shape-law library (slab slope, KV dtype arithmetic, TTM law)
- [[frank-archon-radiance-qwen38-native-fp8]] — the fp8-native era
(rebuild-only now; owns the descriptor-selection law)
- [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here
- [[deployment]] — the service-scoped deploy this cutover rides
serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-native-fp8.md
-308
View File
@@ -1,308 +0,0 @@
# frank-archon — Radiance (StillDeadcode), Qwen3.8-Flash-Next-FP8 NATIVE, 2× R9700 TP=2
**Status: RETIRED 2026-10-05 → 2026-10-08** — replaced by the Swift1.5 w4dn8
conversion ([[frank-archon-radiance-swift15-w4dn8]]), itself retired 2026-10-15;
the payload live today (2026-10-18) is the publisher iq4r container,
[[frank-archon-radiance-qwen38-iq4r]] (`scripts/frank-archon.env:110-111`). ·
**Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 2× Radeon AI PRO R9700 (gfx1201), TP=2
The FreeToken PR #587 endpoint was replaced by the **radiance engine** (C++/HIP,
libr4d kernels for gfx1201, no Python in the serving path) driving the
digest-pinned image `stilldeadcode/radiance@sha256:4c3f484564bb2872784e4…`
(v1.0.7 line, 287 MiB, `libr4d@0.5.0`) as a plain `podman run` through
`scripts/serve-frank-archon.sh` under the **root systemd unit**
(`frank-archon.service`, `User=dank`, `LimitMEMLOCK=infinity`). Same port,
same served alias (`frank-archon`), same container identity
(`frank-archon` = the unit name). Rollback: git tag
`frank-archon-freetoken-pr587-20261005` (FreeToken trio + Dockerfile; the
Dockerfile lives in the tag as `scripts/docker/frank-primary-freetoken.Dockerfile`
— `scripts/docker/` is not in the tree today), image
`localhost/frank-primary-freetoken:pr587-77a7201` (that is the tag on the host —
the 2026-10-10 rename never retagged it), payload
`/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8`, which is **no longer on
disk** (`ls /home/dank/models/` on Dankserver 2026-10-18: `StillDeadcode`, `rad`,
`tpl.jinja` only).
## Why radiance
- **TP=2 is real here.** FreeToken's TP=2 dies in the `pynccl.cu` ROCm JIT
(`cuda_runtime.h` unresolvable for gfx1201 — probed 2026-10-06); the
second R9700 idled. Radiance's TP wire is libr4d's own one-shot
all-reduce (no RCCL), legal on this board since the 2026-09-20 swap
(ProArt B850, x8/x8 on CPU root ports, `Routing+`/`ReqEn+` everywhere).
- **Native FP8 end to end** (Frank's call): the official
`Qwen/Qwen3.8-Flash-Next-FP8` checkpoint keeps its modelopt block-FP8
encoding — no requantisation, no calibration run, no 4-bit codebook.
- **Every tier is explicit and measured**, not an estimator: placement
`expert_tiered` with a pinned host pool, `--ngram-placement ram` for the
47.7 GiB PLE table, `--expert-vs-cache-ratio 0.82` (the publisher's
measured 99 %-hit point; 0.87 already collapses to 0 %), fp8 KV.
## The payload: native FP8, reshaped then converted (one recipe rule)
- **Raw checkpoint**: `/home/dank/models/Qwen/Qwen3.8-Flash-Next-FP8`
(ungated repo, 172.8 GB, 131 shards + index + tokenizer + vision
configs). Fetched by `scripts/download-frank-archon.sh` (vendored
`hfget.sh`, curl+jq, no HF token needed) and **verified byte-for-byte**:
all 131 shards sha256-match the Hub LFS oids (2026-10-05, log
`/home/dank/logs/verify-qwen38-official-fp8.log`).
- **The container**: `/home/dank/models/rad/qwen4exp-fp8-native.rad`
(177,109 MiB measured = 173.0 GiB; floor
`FRANK_ARCHON_RAD_MIN_MIB=177000` in the env gates a
truncated convert). Built by `scripts/build-frank-archon-rad.sh`:
1. stage `/home/dank/models/stage-qwen4exp-fp8` — `config.json` with
**`quantization_config` dropped** + relative symlinks into the raw
checkout;
2. `stack-qwen38-official-fp8.py` reshapes the staging dir — two
transforms, both value-preserving (script docstring carries the full
argument): the plugin maps the STACKED expert layout
(`mlp.experts.gate_up_proj` `[512,1280,2560]` + `_scale_inv` at the
same index) while the official repo ships per-expert unfused tensors,
so staging byte-stacks all 49 stacks' (48 trunk + MTP) 512 experts
with NO requantisation; and the PLE table ships as 128 **scaleless**
E4M3 shards + one bf16 scalar (1.9931793212890625e-04) while the
plugin's fp8 branch wants the codes beside a ONE-ELEMENT bf16 scale
plane — so staging decodes to a single bf16 `shard_0` with the
checkpoint's own scale (e4m3 decode and the scale are both exact in
bf16);
3. `rad-convert` with ONE `--quant` rule — pure copy for everything else
(every weight keeps the checkpoint's block-FP8 planes: `X.weight`
E4M3 codes + `X.weight_scale_inv` bf16 `[128×128]` scales) — and the
rtn rule re-encodes the bf16 PLE table at the same fixed scale, an
exact round-trip (nearest-code distance 2⁻³ vs the bf16 rounding
perturbation 2⁻⁹): the container gets the checkpoint's own PLE codes
plus the one-element scale plane (`blk.1.ple_ngram.scale`, view {1});
4. THREE LAWS the live runs proved (each cost a full verify+declare
cycle; all fixed in the two scripts, 2026-10-05/06):
- **Scale-name law**: `scale_name()` appends `_scale_inv` to the
TENSOR NAME verbatim — a stacked tensor named
`…experts.gate_up_proj` (no `.weight` suffix) has its scale plane
at `…experts.gate_up_proj_scale_inv`, NOT `…weight_scale_inv`.
- **File-shadow law**: the reader opens every index-named file in
SORTED order and emplaces EVERY tensor each file physically
holds, first-wins (`checkpoint.cpp add_shard`). The raw
model-000NN shards carry tensors NAMED exactly what the merged
PLE shard is named, so the merged file MUST sort first
(`00-ple.merged.safetensors`) or a raw fp8 shard wins the name
and the plan reads one shard as the codes.
- **Symlink-through law**: staging symlinks the raw index into the
stage; `open(path,"w")` writes THROUGH it and corrupts the raw
checkpoint (raw verify then fails 143/144). The stack script
writes index/config temp+`os.replace` — an atomic swap replaces
the symlink itself; the build script COPIES the index and
symlinks only the shards.
5. `rad-info` summary + size floor.
- **Why the staging dir is mandatory** (the descriptor law, spec §2.4):
the plugin is selected by the exact `(arch_id, quant descriptor)` pair
with no fallback. A raw official checkpoint carries
`quantization_config.quant_method: "fp8"` → descriptor `"fp8"` → **no
shipped plugin claims `(qwen4exp, "fp8")`** — raw serve AND unmodified
convert both refuse. The shipped `qwen4exp_fp8` plugin claims
`(qwen4exp, "")` (recipe-built containers). `rad_convert.cpp` writes
`w.set_quant(meta.quant)` into the header, so the staged config's empty
quant method flows through as the container descriptor while **every
weight stays FP8 data**. Native precision is untouched; only the
selection key differs.
- `--plan-only` first: the convert's declare phase is the engine's own,
so a plan that resolves every op will boot. The convert runs card-less
(declare = CPU-side layout queries); add `--device` only if a plugin
load ever demands it.
## The shape (`scripts/frank-archon.env`, every value commented there)
| Knob | Value | Why |
|---|---|---|
| Image | `stilldeadcode/radiance@sha256:4c3f4845…` | digest-pinned INDEX digest (RepoDigests also carry the per-arch manifest `1c582005…`; the image ID `69d05724…` is the config digest, NOT a manifest digest — pinning it sends `podman run` to the network); entrypoint = engine, tools (`rad-convert`/`rad-info`/`rad-tune`/`rad-kbench`) on PATH |
| Container | `frank-archon` | identity = service name across runtimes; legacy list retires `frank-archon-qwen frank-archon-r9v qwen-fp8 glm-probe` |
| `--tp 2 --tp-wire wht6` | both R9700s | fused WHT-6 lossy wire: +5.6–6.3 % prefill, ~−1.3 % 8-stream decode at TP2; `--tp-wire-min-kb` stays engine default (128 KiB) |
| `--max-num-seqs 2` | first boot | Frank: "parallelism = 2 to start". Raise only after a real-traffic look at slot occupancy + the ledger's KV headroom line. The fused all-reduce row count is `seqs × (1 + spec) = 8` |
| `--max-model-len 200000` | served window | native 262,144; the publisher's production value and what `models.yml` quotes. Longer prompt = refused, never truncated |
| `--placement expert_tiered --host-pool-mib 40960` | pinned expert pool **per rank** | native FP8 experts are 112.8 GiB total (56.4 GiB/rank slice) vs ~29 GiB at the 4-bit publisher shape — the pool is the price of native precision. Resize from the boot ledger's coverage line, never arithmetic |
| `--gpu-headroom-mib 96 --expert-vs-cache-ratio 0.82` | publisher values | 0.82 = measured 99 % expert-cache hit; 0.87 → 0 % (the cliff) |
| `--kv-cache-dtype fp8` | **7.32 KiB/token** | 12 QSA full-attention layers, 2 KV heads × 256 dim, K+V; a 200K-token sequence is 1.43 GiB. KV is sized from the VRAM the ratio leaves |
| `--num-speculative-tokens 3` | MTP head | the checkpoint carries `mtp.*` (3,101 tensors, `mtp_num_hidden_layers: 1`); the plugin's probe returns draft depth 3 — the publisher's operating point |
| `--max-num-batched-tokens 2048` | prefill chunk | A/B-verified optimum (1024/2048/4096 ladder, see "Tuning levers") |
| `--prefix-cache-host-mib 4096 --prefix-cache-dir /data/kvcache/frank-archon --prefix-cache-disk-mib 65536` | 3 tiers | host tier pinned (see the ledger), disk tier in `/home/dank/radiance-state` (mounted rw at `/data`; `/` is ext4, mapped-tier-eligible); a session snapshot moves with its blocks |
| `--ngram-placement ram` | **PLE pinned to host RAM** | Frank's pin law. The PLE n-gram table is 51.2 GB (320,001,536 rows × 160 dims = 29 % of all parameters) served via io_uring reads; `ram` **refuses the boot** if memlock is too small |
| `--override-chat-template /templates/template.jinja` | froggeric v22.5 | vendored `scripts/qwen38-froggeric-v22.5.jinja` (28,234 B, `qwen3.8-froggeric-v22.5`); rad-convert bakes the checkpoint's own template, this overrides it, and the engine derives the reasoning/tool-call reply format from it |
**Pinned-RAM ledger** (root unit + `LimitMEMLOCK=infinity` exist for this):
2 × 40 GiB expert pools + 47.7 GiB PLE + 4 GiB prefix host tier + ~2 GiB
spare ≈ **134 GiB**. `ttm.pages_limit` (this kernel has NO `pages_max` —
the two-knob story was stale) caps GPU-mappable pinned pages (default = half
of RAM): the pre-cutover cap was **123 GiB** — BELOW the ledger — so the
serve script's TTM preflight (`FRANK_ARCHON_TTM_MIN_MIB=140000`) refuses to
start until it is raised. DONE 2026-10-05 pre-cutover: sysfs write of
**52,428,800 pages = 200 GiB** (`/sys/module/ttm/parameters/pages_limit`;
verified read-back; raising at runtime is harmless and allocates nothing).
PERSISTENCE: the sysfs write does not survive reboot, and the kernel-cmdline
`ttm.pages_limit=52428800` was never added — the unit now raises the knob at
every boot instead (`scripts/frank-archon.service:44`,
`ExecStartPre=+…/scripts/ttm-preflight.sh --raise`).
**Per-rank VRAM** at TP2: expert slice 56.4 GiB total across VRAM + host
pool; trunk ~3.75 GiB/rank; token embedding vocab-parallel (~1.18 GiB/rank
— the bf16 table is 2.43 GiB, sharding hands the other half to KV); lm_head
row-sharded; recurrent state 54 MiB/copy/card (2 copies with speculation);
KV from the `0.82`-side remainder of claimed VRAM at 96 MiB headroom.
## The engine contract (all verified in source, not assumed)
- **`/health` == `/ping`**: 200 JSON (`model_id, in_flight, queue_depth`)
while serving, 503 only while stopping — **the listener binds AFTER the
full load**, so a bare `curl -fsS http://127.0.0.1:7770/health` gate is
honest (the deployer loops on it; the FreeToken-era `"status":"ok"` body
grep is gone). Boot budget 1800 s (publisher `start_period` 15 m);
measured 2026-10-06: health 200 ~2-3 min after start (~120 s warm) —
the big terms are the 125.26 GiB page-in (38.1 s at 3.5 GB/s) and the
host-pool pin (PLE 47.68 GiB in 43.9 s).
- **Sampling** (oai.cpp): `temperature, top_p, top_k, min_p, typical_p,
presence_penalty, frequency_penalty, repetition_penalty, repeat_last_n,
chat_template_kwargs, reasoning_effort, enable_thinking,
preserve_thinking` accepted; **unknown params are refused (400)** —
models.yml's extraBody set is fully inside it; nothing unverified rides
the wire. `max_tokens` is clamped, not refused; `seed` absent = pick
one; `response_format` supported; **`logprobs` is refused** (harness
never sends it).
- **Reasoning**: `--reasoning-format` defaults to `auto` → thinking in
`reasoning_content` (models.yml `reasoningContentField` matches). Tool
calls: the qwen3.8 dialect → `message.tool_calls` +
`finish_reason: tool_calls`. No parser flags anywhere — parsers are
auto.
- **Vision**: the official checkpoint carries the 27-layer vision tower;
the engine runs the Qwen-VL processor (patching, pad-token expansion,
3-part rotary are core). `models.yml` keeps `input: [text, image]`;
`--mm-max-patches` available.
- **`/metrics`**: vLLM-compatible names + `radiance:*` families — the
Dankclaw Prometheus job re-scrapes 192.168.0.235:7770 (plain LAN routing
reaches it; the netavark published-port quirk is same-host only).
`/stats`, `/sessions`, `/dashboard` for live internals; the repo ships
`deploy/grafana/radiance.json` if a dashboard is wanted.
- **`n` > 1 requests**: `n` defaults to `max(8, max-num-seqs)`; stop
strings ≤ 64 × 4096 B. `model` field not enforced.
## Deploy + verify
```bash
# one-time (already done 2026-10-05): payload fetch + sha256 verify + convert
ssh dankserver 'bash /home/dank/frank-archon-build/build-frank-archon-rad.sh --plan-only' # declare + plan, no bytes
ssh dankserver 'bash /home/dank/frank-archon-build/build-frank-archon-rad.sh' # full convert + rad-info
./scripts/deploy.sh dankserver --service frank-archon
```
Post-cutover checklist (each is observable, not assumed):
1. Unit: `is-active` active, `is-enabled` enabled, container answers
`/health` with 200 JSON.
2. Boot ledger (engine stdout → `/home/dank/logs/frank_archon.log`):
P2P probe per rank pair, TP=2 shard placement, expert pool coverage
line ("backs N % of the worst case … M full-length sessions"), KV pool
bytes, PLE pinned-RAM line, drafter depth 3, prefix tiers.
3. MTP live: a completion's `timings` object shows draft/accept counts;
`radiance:*spec*` metrics in `/metrics`.
4. Froggeric template: a probe prompt renders the v22.5 shape
(tool-call + thinking turn both exercise it).
5. Vision: a ≥256 px solid-color PNG answers the exact color through an
`image_url` content part.
6. Prometheus target `frank-archon` green (not never-green).
## Cutover measured (2026-10-06, all values observed live)
- Container: 1,857,127,997,440 B = 177,109 MiB = 173.0 GiB. Plan-only
priced 172.76 GiB — the DATA planes; the written container adds a
metadata tail. Floor gates must be set from the real file (the first
deploy refused on 178000 > 177109; floor is 177000).
- Boot: 125.26 GiB read in 38.1 s (3.5 GB/s), PLE pinned
"47.68 GiB in 43.9 s" (the 2 B scale plane with it), each rank's
mover holds 25.46 GiB static+experts (19.15 GiB resident experts,
79 %), 16,820 pinned pool slots, health 200 ~2-3 min after start
(~120 s warm). No 7-minute vLLM-style boot.
- KV ledger: kv_attn 99,956 blocks = 2.48 GiB (399,824 tokens),
kv_qsa_bkey 317.24 MiB, kv_gdn_state 4 slots × 54 MiB, PLE conv
states < 1 MiB; linear checkpoints 56.37 MiB × 8 slots (3 of 8
card-backed at boot, 5 withdrawn under the 208 MiB floor).
- MTP on real traffic (temp 0.7, 220-token ask): draft_n 225 /
draft_n_accepted 119 = 52.9 % acceptance, decode 23.8 t/s. ONE
sample — the shape's real number needs live-traffic measurement.
- Contracts: `reasoning_content` field (matches models.yml), usage
metering separates reasoning_tokens; vision — a 512² solid-red PNG
through `image_url` answers `Red` (279 prompt tokens; NOTE the
32-token budget is eaten by thinking — give vision probes ≥256
max_tokens); native `tool_calls` with finish_reason tool_calls.
- Boot defects fixed the same day: the SSD prefix tier opened only
after the serve script mkdir -p'd the full path (the engine's
create is NOT recursive: "cannot create /data/kvcache/..."); the
converter's scale-name law, the PLE file-shadow and the
symlink-through-raw-index bug are in the payload section above.
- Prometheus: `/metrics` answers 200 with vLLM-compatible families
(verified from Dankclaw); the `frank-archon` job needs a
prometheus-service deploy to go green (config pushed, reload
deferred behind a live parallel writer's dirty tree).
## Tuning levers (in order of likely use)
- **`FRANK_ARCHON_MAX_SEQS`**: raise 2 → 3/4 only when the ledger's KV
headroom line and real slot occupancy agree; each slot buys a
recurrent-state copy (54 MiB/card) and shrinks the per-request KV share.
- **`FRANK_ARCHON_HOST_POOL_MIB`**: the ledger's coverage line is the
oracle — it prints what the pool backs at the worst case and how many
full-length sessions fit. RAM budget: pools + PLE + prefix ≤ ~140 GiB of
the host's 246.
- **`FRANK_ARCHON_EXPERT_VS_CACHE`**: 0.82 is the measured 99 %-hit
point; 0.87 zeroes it. Move only with a measured hit-rate reason.
- **`FRANK_ARCHON_TP_WIRE`**: `wht6` vs `exact` — `exact` trades the
+5.6 % prefill for bit-exact reductions; the fused one-shot forms are
declared over the seqs × (1+spec) row bound.
- **`FRANK_ARCHON_EXTRA_ARGS`**: the boot-ladder escape hatch, expanded
unquoted.
- **`rad-tune -m <rad> --tp 2 --num-speculative-tokens 3`**: benchmarks
the tunable space at this model's shapes on this machine, writes
`$RADIANCE_HOME/tune/<machine>.tune` which the engine's selector reads at
boot. Declare at the shapes you serve (`--tp 2
--num-speculative-tokens 3`), or you measure a program the serve never
runs. `$RADIANCE_HOME` is `/opt/radiance/share/radiance` — in the IMAGE — so a
`--rm` scratch container's cache dies with it: the serve script
bind-mounts `$FRANK_ARCHON_STATE_DIR/tune` (`/home/dank/radiance-state/tune`)
at `/opt/radiance/share/radiance/tune`, so the cache (36 GEMM rows,
`gfx1201-x3-AMD_Radeon_Graphics.tune`, generated 2026-10-06 in a
stopped-service window) survives and is read at every production boot.
The two attention requests have no shape recipe (`attn_paged`,
`attn_paged_gate_quant` — "see tools/opshapes.h") and are not tuned.
`RADIANCE_TUNE` pins a kernel for an A/B — an instrument, not a setting.
- **`FRANK_ARCHON_MAX_BATCHED_TOKENS` (MNBT)**: A/B'd 2026-10-06 in a
stopped-service window (real-text probes: 20,417-token cold prefill,
1,985-token warm prefill, 256-token decode). The prefill ladder is
NON-MONOTONIC with an interior optimum at 2048: 1024 → 2048 → 4096 =
231 → 460 → 448 t/s cold, 221 → 412 → 427 t/s warm. Decode is flat within
noise (24.0 / 25.1 / 24.3 t/s); draft acceptance 60.1 / 67.1 / 63.3 %.
**2048 wins** (peak prefill + peak acceptance) — it stays the production
value. 1024 halves prefill (the batch too small to saturate the card);
4096 is a hair off 2048 (the chunk past the kernel's sweet spot).
- **Diagnostics**: `rad-info` (container: encodings, recipe, profile),
`--debug-graph` (what declared vs what issued), `rad-kbench` (kernel
correctness/speed vs libref), `/stats` + `/sessions` (live), and the
publisher's `scripts/` harnesses (`fnpf`, `fnconc`, `cstep2`, `ident`,
`mctools`) — note `mcserve/fnstop`-family scripts `pkill` the engine
binary by name; on this host the service is a systemd unit, so use
`sudo systemctl stop frank-archon` instead.
## Rollback
| Layer | Path |
|---|---|
| FreeToken (one card, native FP8 safetensors) | `git checkout frank-archon-freetoken-pr587-20261005 -- scripts/frank-archon.env scripts/frank-archon.service scripts/serve-frank-archon.sh scripts/docker/ tests/frank_archon_unit.bats` + redeploy; the image in storage is `localhost/frank-primary-freetoken:pr587-77a7201`, and the payload `/home/dank/models/Qwen3.8-Flash-Next-Uncensored-FP8` must be re-fetched — it is gone from the host |
| tcclaviger vLLM fork (TP=2, shape G) | git tag `frank-archon-vllm-shapeG-20261003` (trio + env lore; image still in dank's rootless storage) |
| 4-bit StillDeadcode publisher shape | `qwen3.8-next-flash-fp8-iq4r-moe.rad` — **this became the live payload on 2026-10-15** ([[frank-archon-radiance-qwen38-iq4r]]), deployed with `--host-pool-mib 44928` (`scripts/frank-archon.env:184`), not the 12,288 MiB first sketched here |
The FreeToken-era orcarouter payload has since been deleted from the host
(`ls /home/dank/models/` on Dankserver, 2026-10-18) — there is nothing left to
trash, and the FreeToken rollback needs a re-fetch.
## Links
- [[LLM/index]] — fleet index
- [[frank-archon-freetoken-pr587]] — the retired FreeToken era
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the retired tcclaviger vLLM era
- [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here
- [[RDNA4 Container Serving]] — rootless-podman GPU laws on these hosts
- `serverAppConfigs-docs/Architecture/Testing Policy.md` — the seam contract
serverAppConfigs-docs/LLM/frank-archon-radiance-swift15-w4dn8.md
-625
View File
@@ -1,625 +0,0 @@
# frank-archon — Radiance (StillDeadcode), Swift1.5 Qwen3.8-Flash-Next W4Dn8, 2× R9700 TP=2
**Status: RETIRED — superseded 2026-10-15 by the publisher iq4r payload.** · Lived 2026-10-08 → 10-15 (main `3ca64e1`, e2e 33/33). · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 2× Radeon AI PRO R9700 (gfx1201), TP=2
The FP8-native era (`qwen4exp-fp8-native.rad`, 173.0 GiB, no-recipe convert)
is replaced by a **GPTQ-calibrated 4-bit container** of
`ukisai/Swift1.5-Qwen3.8-Flash-Next` (BF16, 335.3 GiB, abliterated
finetune) converted with the recipe
`scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe` on engine
**1.0.13** (`stilldeadcode/radiance@sha256:4c3f484564…` — same digest as
the FP8-native era; 1.2.0 was used ONLY for the calibration serve).
Same port, alias, container name, systemd unit, and every serving argument
carried into the successor: [[frank-archon-radiance-qwen38-iq4r]]. The
`.rad` itself was purged from disk 2026-10-17; the byte-exact payload
lives on the Hub — see Rollback.
## Why this swap
- **Context economics at production memory budgets**: the FP8-native
experts (56.4 GiB/rank slice) forced a 40 GiB/rank pinned host pool and
left KV ~3.5 GiB. W4Dn8 experts cut the resident set to ~1/4, freeing
host and VRAM budget for KV and slots at equal-or-better measured
decode (gate: ≥ 30.8 t/s vs the 30.08 t/s control).
- **Swift 1.5 base**: abliterated finetune of Flash-Next (Frank's
fleet-personality call; frank-dragoon serves the same family).
- **GPTQ closes the precision gap**: RTN alone on BF16 weights lands
measurably worse; GPTQ with Hessians from a 10M-token corpus of this
model's own distribution recovers most of it, verified end-to-end by a
KLD gate against the BF16 container itself (KL ≤ 0.03, top-1 ≥ 95 %).
## The payload chain
1. **Fetch**: `scripts/download-frank-archon-swift15.sh` →
`/home/dank/models/Swift1.5-Qwen3.8-Flash-Next` (131 shards, 335.3
GiB, per-shard LFS sha256 verified, no token on the command line).
48 layers, 512 experts (top-10 routed), moe_inter 640, hidden 2560,
PLE 20M-vocab table 95.37 GiB, MTP head, vision tower.
2. **BF16 container**: `run-swift15-bf16.sh convert` →
`rad-convert` no-recipe (pure copy) at `--max-ctx 4096` →
`qwen4exp-swift15-bf16.rad` (335.28 GiB). Serves on 1.0.13 as the
KLD reference engine (disk-tier placement; boot ~35 min).
3. **KLD reference**: `run-swift15-bf16.sh kld` (1.0.13) →
`/home/dank/corpora/kld-ref` — `radiance-kld-1` format, `rows:
50235`, 197 source-labelled docs, top-16 log-probs (f16), corpus
fnv1a64 `a9d62b33c18b3d9c`. ~4 h at 32k prefill steps
(`--max-num-batched-tokens 32768` — a 65536 step OOMs an arena
tensor at `--max-model-len 16384`). **The wrapper's exit line is the
receipt: never scp this script while it runs** (bash lazy-read;
a mid-run swap killed a clean engine exit with rc=127, third fleet
occurrence of the script-swap law). `kld-then-calib.sh SKIP_KLD=1`
resumes from the `rows: 50235` receipt.
4. **Calibration**: `run-swift15-bf16.sh calib` on **1.2.0** (the only
engine with `RADIANCE_CALIB_TOKENS`; grams are engine-invariant
bf16 activations — RADGRAM2 format byte-identical across versions)
+ `calib-push.py` pushing `calib-corpus.jsonl` (10,014,932 tokens,
1,294 prompts, assembled from real HF sources, chat-templated,
source-labelled). The tap (in the `qwen4exp_fp8.so` architecture
plugin) writes per-expert down-proj Gram matrices
`gram.r<rank>.blk.L.ffn_down_exps.<e>.bin` (32-byte RADGRAM2 header +
640×640 f32; `rows` = routing count, `trace` = activation energy)
and stops the serve at the token threshold. TP=2: whole experts per
rank (256/rank); gate_up input is rank-sliced (plugin warning — the
recipe only casts the down projection).
5. **Protected-expert derivation**: `derive-protected-experts.py
/home/dank/corpora/calib` → the 24 highest-energy experts (the law
is data-driven; Swift drew 24, not the fp8-era 10 — its routing is
spikier: `blk.47 e445` alone carries 32 % of its layer's down energy
at 70× routing share) promoted to whole-expert bf16.
Protection law: energy share ≥ 3 % AND ≥ 3× routing share; even count
per rank half; odd half gains the next most energetic.
`apply-protected-cast.sh` splices the derived names between the
`# PROTECTED-BEGIN/END` markers of the recipe (refuses double-derive).
6. **Final convert**: `convert-swift15-w4dn8.sh` (1.0.13, host-backend
CPU-only — no GPU visible to `rad-convert`; GPTQ solve is CPU BLAS) —
gates: ≥48 gram files present, bf16 container already deleted
(disk), ≥146 GiB free, output ≥108,000 MiB, `rad-info` shows the
gptq receipt. No `--reuse`. Measured: 5 h 50 m (6.0–6.5 min/layer on
49 GPTQ layers, load-average 8), output **116,416 MiB**.
The plugin's `rad_arch_declare` enforces ONE ENCODING PER EXPERT
TABLE (`one table of experts takes one encoding, and only a whole
expert kept plain bf16 is served beside it`) — a split tier
(down=fp8, gate_up=w4) is unservable: the kernel serves a single
w4nl64a8h table; both expert tables ride GPTQ w4 (down calibrated,
gate_up uncalibrated), protected experts bf16 beside them.
## The recipe tiers (scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe)
| Tier | Encoding | Why |
|---|---|---|
| MoE experts, both `gate_up` and `down` (one w4 table/layer) | **GPTQ w4nl64a8h** — u4 codes, fwht128 rotation, fp8 scales/64 + f32 scale plane | the campaign tier: GPTQ on down-input Hessians from the 10M-token corpus (the tap rides the down projection; gate_up shares the table uncalibrated — one-encoding-per-table plugin law) |
| 24 derived protected experts (whole) | **bf16** | energy-share law (step 5) |
| shared expert + trunk attention/SSM linears | **int8**, scale per 128 cols, searched | trunk precision law |
| hyper-cond + MTP head | **fp8_e4m3 row-groups** | hc+mtp fp8 law |
| lm_head | **int8** | i8 lm_head law (E4M3 costs 0.6 top-1 alone) |
| PLE n-gram table | fp8 at the checkpoint's fixed scale 1.9931793212890625e-04 | the fixed-scale law; absmax audit passed the 2⁻¹³..2⁻¹⁰ plateau |
| embeddings, vision tower, QSA indexer | **bf16 passthrough** | untouched |
## The shape (`scripts/frank-archon.env`)
TP=2 + `wht6`, `--max-model-len 200000`, MNBT 2048 (the MTP resonance law —
2560 collapses decode to 2.6–7.7 t/s), `--ngram-placement ram`, froggeric
v22.5 template override. `MODEL_FILE` points at
`/home/dank/models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416 MiB
measured**); the serve size floor `FRANK_ARCHON_RAD_MIN_MIB` moves
177000 → **110000**. Concurrency and the KV cache moved on 2026-10-11; that
change is documented in its own section below.
## 2026-10-11 — unquantized KV at 2 × 200,000
The ask: two concurrent 200,000-token conversations on exact (unquantized)
attention state. The fp8 pool backed 3,543 MiB → **386,904 tokens = 1.93
sessions of the served window**, so two full-height conversations overflow it
and the scheduler preempts the overflow to the host/disk prefix tiers
(re-prefilling on return). Measured, that overflow is survivable — the fp8
shape passed the same 2 × 197k pair probe — so this is a **headroom** change,
not the repair of a hard failure. The hard limit the engine states is per
request, not per pair: a request longer than the pool's token count is
*"accepted by the API and then refused as too large to ever fit — this is a
hard limit, not a backing ratio"*, and at 200,000 ≤ 386,904 no single request
ever hit it.
### The dtype law
`--kv-cache-dtype` takes **only `bf16 | fp8`**: one `kv_dtype` for the whole
cache. llama.cpp's exact-K / quantized-V pair (what frank-zealot runs as
`--cache-type-k f16 --cache-type-v q8_0`) has **no radiance equivalent** — the
rung under bf16 is fp8 on both halves. Evidence: the flag table in the image
itself (`--kv-cache-dtype takes bf16 or fp8, not '%s'`) and the absence of any
`cache-type-k`/`-v` symbol in the binary.
bf16 costs **1.89× fp8, not 2×**, and the reason is in the group table:
| Group | fp8 page | bf16 page | cost per token, per rank |
|---|---|---|---|
| `kv_attn` — 13 full-attention layers, 4-token blocks | 26.00 KiB | 52.00 KiB | 6.5 → 13.0 KiB |
| `kv_qsa_bkey` — 13 layers, the QSA indexer keys | 3.25 KiB | **3.25 KiB** | 0.8125 KiB, dtype-fixed |
| `kv_gdn_state` 54 MiB × 2 copies per sequence (+ conv/tail/ple) | per slot | per slot | ≈ 110 MiB **per sequence** |
| prefix checkpoints 8 × 55.99 MiB (inside the pool) | 448 MiB | 448 MiB | fixed |
→ `pool_mib ≥ tokens × 13.8125 KiB + 110 × seqs + 448`. The formula reproduces
both live boots: fp8 `(3543 − 330 − 448) → 387,240` vs the ledger's 386,904;
bf16 `(6912 − 220 − 448) → 462,946` vs the ledger's **462,752**. It is encoded
as a test seam (`the KV pool backs seqs x max-model-len at the declared cache
dtype`) so a future resize cannot silently under-provision the pool.
### Why the weights slab pays — and why that is not pure loss
The two VRAM budgets are one budget, and the ceiling is the engine's own — not a
hand-picked reserve. Card 32,624 MiB, and the table that prints at the pair
24,000/5,120 says `3.42 GiB is neither budgeted nor accounted — the driver's
context, the activation arena and --gpu-headroom-mib come out of it; anything
past those is throughput on the floor`, with `card 0 settled at 2.01 GiB free
against 209.19 MiB (the 96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the
static region)`. Claimable reserve above the ENGINE's floor:
2,058 − 209 = 1,849 MiB ⇒ **`weights + kv ≤ 30,969 MiB`**. (The 29,120 bound
written on 2026-10-11 was my own 1 GiB "keep free" reserve, measured off a card
figure of 31,856 MiB; the engine never asked for that reserve and the seam now
derives the number from its table.) Paying for a 6,912 MiB pool therefore cuts
the resident slab 26,071 → 22,208 MiB (−14.8 %), and the evicted experts spill
into the pinned host pool (40,960 → 44,928 MiB per rank; ledger: pinned pool
slots 7,190 → 10,375). The pinned-RAM ledger rises to ≈147 GiB, so the TTM
floor moved 140,000 → 150,000 MiB (the kernel cap is 204,800).
Half of that loss is refunded by an engine behaviour worth knowing: **idle KV
VRAM is the expert cache's spillover**. The fp8 pool lent 2.69 GiB to the expert
slab; the bf16 pool lends **6.08 GiB** (`mover: 5146 slots can be lent out of
the KV cache`). On a llama.cpp endpoint KV is inert storage; on radiance it is
a competing cache, and every KV resize is also an expert-residency resize.
### seqs 3 → 2
`--max-num-seqs 3` was never a radiance requirement: it is the tcclaviger vLLM
fork's torch.compile constraint (`ConstraintViolationError: Constraints
violated (L['query_start_loc'].size()[0])`), retired with that runtime — see
[[dankserver-qwen38-fp8-expert-offload-tuning]]. Radiance booted seqs 2 in the
fp8-native era, and the 2026-10-06 decode sweep measured seqs 2 and 3 tied
inside the ±3 % noise floor. Two slots × the window is the capacity contract;
it also saves 110 MiB of per-slot recurrent state and drops the fused TP
all-reduce to 2 × 3 = 6 rows, i.e. deeper inside the 25-row bound at which the
wht6 wire is bit-exact.
### Boot ledger (deployed 2026-10-11, commit `0bf26b0`)
```text
kv cache 6.75 GiB 349% of it -- backs 2 sequences all at 200000
tokens at once, which is the worst case
vram_weights 21.69 GiB stated by --vram-weights-mib
vram_kv 6.75 GiB stated by --vram-kv-mib
unclaimed 2.03 GiB
kv_attn full 13 layers block 4 tok page 52.00 KiB 115688 blocks = 5.74 GiB (462752 tokens)
kv_qsa_bkey full 13 layers block 4 tok page 3.25 KiB 115688 blocks = 367.17 MiB (462752 tokens)
load: rank 0, mover took 21.69 GiB, weights pool 147.00 KiB free of 21.69 GiB
mover: 5146 slots can be lent out of the KV cache (6.08 GiB)
card 0 settled at 1.85 GiB free
serving on http://:::7770
```
### Verification (live service, no scratch container)
| Check | Result |
|---|---|
| `/health`, `/v1/models` | 200 `{"status":"ok","model":"frank-archon"}`; served id `frank-archon` |
| 2 × ~197k concurrent, a unique marker buried at the head of each | **both answered their own marker exactly** — 196,919 + 196,918 = 393,837 tokens resident, above the fp8 pool's 386,904 and inside bf16's 462,752; walls 138.1 s / 66.2 s, `finish_reason: stop`, prefill 2,747 / 2,991 t/s |
| decode at 51.5k depth, 512 tokens, greedy | 94.1 cold / 102.3 warm / 117.6 on a prefix-cache hit t/s; draft acceptance 0.7284 / 0.7525 (fp8 control at cutover: 0.7246) |
| fp8 control arm, same probes (`scripts/probe-frank-archon-kv.py`) | table in the next section |
Probe law re-learned, worth recording: radiance routes the thinking block to
`reasoning_content`, so a chat request whose budget is spent inside `<think>`
comes back with **empty `content` and `finish_reason: stop`** — a probe that
forbids thinking (`chat_template_kwargs.enable_thinking=false`) and reads both
fields is the only honest shape. The first pair probe scored 0/2 on exactly
that artifact, not on a KV fault.
### Measured cost — A/B on the live service
Method (no scratch container, no mask — the unit file is a relink symlink and
`mask` refuses it): write a variant env file on the host
(`/home/dank/frank-archon-fp8control.env`), point a transient drop-in at it
(`Environment=FRANK_ARCHON_ENV_FILE=…`; drop-ins are read AFTER the unit's own
`Environment=` lines, so they win), `daemon-reload`, restart, probe, delete the
drop-in, restart. Control = the deployed 2026-10-08 shape, verified from
`podman inspect` argv (`--kv-cache-dtype fp8 --vram-weights-mib 26071
--vram-kv-mib 3543 --max-num-seqs 3`) and the ledger page size (26.00 KiB).
Identical probes, same salts, loopback, greedy, 512 generated tokens.
| Probe | fp8 control (26071/3543, seqs 3) | bf16 deployed (22208/6912, seqs 2) | Δ |
|---|---|---|---|
| prefill 51.5k, cold | 4,379 t/s | 3,126 t/s | **−29 %** |
| prefill 51.5k, warm | 4,864 t/s | 3,333 t/s | **−31 %** |
| decode 51.5k, cold | 124.1 t/s | 94.1 t/s | −24 % |
| decode 51.5k, warm | 99.2 t/s | 102.3 t/s | +3 % |
| decode on a prefix-cache hit | 127.7 t/s | 117.6 t/s | −8 % |
| draft acceptance, mean of passes | 0.661 | 0.740 | **+12 %** |
| 2 × 196,919 concurrent pair | both markers HIT, 110.5 s | both markers HIT, 138.1 s | +25 % wall |
Read it as: **the price of exact KV on this engine is prefill, not decode.**
Prefill is expert-bandwidth-bound and the resident slab is 14.8 % smaller, so
time-to-first-token at the full window rises ~25 % (the pair: 110 s → 138 s);
both prefill passes agree, far outside the ±2 % prefill noise floor. Decode is
inside the ±3 % band on two of three passes (warm +3 %, hit −8 %); the cold-pass
−24 % is the largest single number, the least reproducible, and it moves the
same direction as prefill — the cold arm is paying expert streaming, not
attention. Spec-decode acceptance IMPROVED ~8 points (0.661 → 0.740): exact
attention state makes the target agree with the MTP drafter more often, which
pays on every token of every conversation, and it is the strongest argument for
the shape.
Open call (Frank): keep exact KV at the prefill cost, or take the cheap fp8
variant — `--vram-kv-mib 3,648` + `--max-num-seqs 2` fits 2 × 200,000 at fp8
for a 105 MiB pool bump and ≈ no decode cost. The tool is committed:
`scripts/probe-frank-archon-kv.py` (run it on Dankserver; it rebuilds its
prompts from the repo checkout there).
### Rollback
Revert `scripts/frank-archon.env` to the fp8 pair (26071/3543, fp8, seqs 3,
host pool 40960, prefix host 4096) and redeploy: no payload change, boot ~2 min.
The KLD gate scored at the fp8-KV shape, so a bf16-KV production runs *away*
from the scored configuration — bf16 can only reduce the attention-state
error, and a re-score is a drained-window job if the gate agreement is ever
needed in the record.
## 2026-10-13 — the power failure, the ttm reset, and 4 × 200,000 fp8 KV
### The outage: a refusal, not a crash
Dankserver lost power on 2026-10-12 ~23:17. The box came back; frank-archon
never did. The unit sat in `activating (auto-restart)`, exiting 1 about 50 ms
after starting, with one journal line on repeat:
```text
ERROR: ttm pages cap is 125988 MiB, the pinned ledger needs >= 150000
frank-archon: raise /sys/module/ttm/parameters/pages_limit …
```
**6,919 refusals in ~20 hours.** No container was created (`podman ps -a` showed
the name with a zero start time), so there was no engine, no port, and no engine
log — the signature of a preflight refusal, which every deployer/monitor view
outside the journal renders as an unexplained crash loop.
The cause is a class, not an incident. The pinned-RAM ledger (~147 GiB: 2 × 44 GiB
expert pools + the 47.7 GiB PLE table + the 8 GiB prefix host tier) rides TTM's
shared pages, capped by `/sys/module/ttm/parameters/pages_limit`. **That knob is
runtime state, and its default is half of RAM** — so the manual raise of
2026-10-05 (52,428,800 pages = 200 GiB) died with the power failure and the cap
fell back to 32,253,039 pages = 125,988 MiB, 24 GiB below the floor the serve
script enforces. The fp8-native era had planned the persistent half of this
("add `ttm.pages_limit=52428800` to the kernel cmdline at the next scheduled
reboot", [[frank-archon-radiance-qwen38-native-fp8]]); it was never applied, and
a power failure is not scheduled.
### The repair: the gate raises the knob itself
`scripts/ttm-preflight.sh` (sourced by `serve-frank-archon.sh`) reads the cap as
the MIN over the knobs the running kernel exposes, and when it is below
`FRANK_ARCHON_TTM_MIN_MIB` it writes `FRANK_ARCHON_TTM_TARGET_MIB` (204,800 MiB =
52,428,800 pages) into every writable knob, **re-reads**, and only then refuses.
The unit is root and the knob is a 0644 root-owned module parameter, so the write
is legal; a kernel that refuses the move produces the same named refusal as
before, and a reboot is now a boot rather than a visit to the machine.
| Knob | Power-failure default | After the gate |
|---|---|---|
| `/sys/module/ttm/parameters/pages_limit` | 32,253,039 pages = 125,988 MiB | 52,428,800 pages = 204,800 MiB |
Seam-tested, not inspection-tested: *the ttm gate raises a reset pages cap and
refuses when the write fails* and *the ttm cap counts only the knobs the kernel
exposes* drive the gate against a temp knob directory — a below-floor writable
knob ends at the target and the start passes; a read-only knob leaves the cap
untouched and the start refuses by name; an absent `pages_max` is skipped, never
read as zero (a zero reading would refuse a healthy host).
### The shape: 4 slots × 200,000, fp8 KV (same day, superseded by the re-cuts below)
Answers the open call of 2026-10-11 from the other side: Frank asked for **four**
concurrent 200,000-token conversations with quantized KV.
fp8 is not a preference at four slots, it is the only dtype that affords them.
The pool must back `seqs × max-model-len` (the engine's hard limit), so the
fourth slot is bought in KV, out of the one per-card budget
(`weights + kv ≤ 29,120 MiB`, the bound as it was understood then):
| dtype | pool for 4 × 200,000 | weights slab left |
|---|---|---|
| bf16 | 10,791 + 440 + 448 = 11,679 MiB | 17,441 MiB (−33 % vs the fp8-era 26,071) |
| **fp8 (deployed)** | 5,713 + 440 + 448 = 6,601 MiB | **22,208 MiB — unchanged** |
So `--vram-weights-mib 22208` / `--vram-kv-mib 6912` stay exactly where the
2026-10-11 change put them (pair sum = the measured 29,120 ceiling), the
44,928 MiB/rank pinned host pool is untouched, and the RAM/TTM ledger does not
move. What the fp8 pool buys: `(6912 − 110×4 − 448) × 1024 / 7.3125` =
**843,566 predicted tokens** = four full-height conversations with 5.4 % slack
(the bf16 pool backed 462,752 = two; the 2026-10-08 fp8 pool backed 386,904 =
1.93).
Terms that move with the dtype and the slot count:
- **Per-slot recurrent state** is dtype-independent and lives inside the pool:
+110 MiB per slot, 440 MiB for four (already in the arithmetic above).
- **The fused TP all-reduce** is declared over `seqs × (1 + spec)` rows: 4 × 3 =
**12 rows**, still inside the 25-row bound at which the `wht6` wire is
bit-exact — so four slots cost nothing in wire exactness.
- **Prefix-cache entries shrink** with the attention plane (110.50 KiB at bf16 →
58.50 KiB at fp8), so the 8 GiB host tier parks ~5.8 sessions of 200k where at
bf16 it parked ~2.9 — four live slots plus a parked set fit with no re-size.
- **`models.yml` is untouched**: `contextWindow` quotes the per-request window
(200,000), which did not change; only the concurrency did.
Expectation carried from the 2026-10-11 A/B, whose arms bracket this shape
(same slab 22,208): prefill is set by the slab, not the dtype, so ~3.1k t/s
cold / ~3.3k warm; decode sits in the 94–118 t/s band with draft acceptance
~0.73–0.75. The 4,379 t/s fp8 prefill of the 2026-10-08 control belongs to the
26,071 MiB slab and does not come back by quantizing KV.
### Boot ledger (2026-10-13 19:21, four slots, fp8 — superseded same evening)
The sizing law's prediction, checked against the engine's own arithmetic:
predicted 843,566 tokens, ledger **843,276** (0.03 % block rounding).
| Group | Page / state | Size | Tokens |
|---|---|---|---|
| `kv_attn` | 13 layers, 4-token blocks, **26.00 KiB** (fp8 page) | 5.23 GiB | **843,276** |
| `kv_qsa_bkey` | 13 layers, 4-token blocks, **3.25 KiB** (dtype-fixed) | 669.10 MiB | **843,276** |
| `kv_gdn_state` | 36 layers, 54.00 MiB, **8 slots** (4 × 2 copies) | 432.00 MiB | per-slot |
| `kv_gdn_conv` / `kv_ple_*` | 36 + 1 + 1 layers, 4 slots | 7.91 MiB | per-slot |
`kv cache 6.75 GiB … backs 4 sequences all at 200000 tokens at once, which is
the worst case` — the capacity contract, in the engine's words. The rest of the
boot is the fp8-era picture: `vram_weights` 21.69 GiB filled to 147 KiB (rank 0)
/ 355 KiB (rank 1) of slack, `unclaimed` 2.03 GiB, cards settled at 1.99 /
2.00 GiB free, `mover: 4967 slab slots (64 shadow), 10375 pinned pool slots,
4.24 GiB static vram`, and the half-refund in plain sight: `4967 slots can be
lent out of the KV cache (5.87 GiB)` (the fp8-era 3,543 MiB pool lent 2.69 GiB).
Two ledger lines that confirm the derived consequences: the sampler prints
**12 rows** (4 sequences × (1 + 2 speculative) — inside the 25-row bound where
`wht6` is bit-exact), and the prefix/tier arithmetic runs on the fp8 entry size
(58.50 KiB, half of the bf16 entry).
### The same-day re-cuts: 4 → 3 slots, then the slab takes the card's slack
Frank's call, twice in one evening. Both are one-env-pair changes, and both were
read off the boot ledger rather than assumed.
**4 → 3 slots (22208/6912 → 24000/5120).** Three slots need
3 × 200,000 at fp8 = 4,285 + 330 + 448 = **5,063 MiB** of pool; the 6,912 pool
was backing 858,969 tokens — 43 % more window than the contract asks for.
Cutting it to 5,120 (predicts 608,027; ledger **607,740**, 0.05 %) hands
1,792 MiB to the slab and drops the all-reduce to 9 rows.
**24,000 → 25,600 (the card's settling slack).** The 24,000 boot printed
`card 0 settled at 2.01 GiB free against 209.19 MiB (the 96.00 MiB
--gpu-headroom-mib and 113.19 MiB kept for the static region)` — 1,849 MiB above
the ENGINE's own floor, with the budget table calling the surplus `throughput on
the floor`. Claiming 1,600 MiB of it puts the pair at 30,720 and the ledger at
`unclaimed 477.30 MiB` (cards ~336 MiB free, floor 209.19).
| pair | slab slots (rank 0) | pinned pool slots | A-cold 43.5k | concurrent 197k |
|---|---|---|---|---|
| 22208/6912, seqs 4 | 14,753 | 10,375 | 3,397 t/s | 3,307 t/s (4-way) |
| 24000/5120, seqs 3 | 16,235 | 8,893 | 3,290 t/s | 3,051 t/s (3-way) |
| **25600/5120, seqs 3** | **17,548** | 7,580 | **3,584 t/s** | **3,589 t/s** (3-way) |
| 26071/3543, seqs 3 (2026-10-11 control) | — | 7,190 | 4,379 t/s | — |
The claim shipped with the 25,600 arm — that the slab is a THRESHOLD below
~26 GiB so the extra 1.6 GiB buys ~0 — is **falsified by its own measurement**:
+8.9 % single-stream, +17.6 % concurrent, both outside the ±3 % noise floor. Two
slab points that land inside noise of *each other* (22,208 vs 24,000) are not a
plateau; the pinned host pool is the slow path, so every MiB moved from RAM onto
the card is throughput (the ledger shows the transfer directly: slab
+1,313 slots, pinned pool −1,313).
The cliff is real, it just sits above 25,600: 26,071 measured 4,379 t/s, +22 %
for +471 MiB. At three slots that is out of reach (26,112 + 5,120 = 31,232 > the
30,969 ceiling). At **two** slots the pool falls to 3,072 and the slab reaches
**27,897** — the ceiling exactly, slab above the cliff. That arm is untested and
it is a capacity decision, not a tuning one.
Capacity contract, live-verified on the 3-slot shape: three concurrent prompts
of **199,984 tokens** each (599,952 tokens) were accepted and completed on
`/v1/completions`; a fourth full-height request is refused as "too large to ever
fit" — radiance never truncates.
### The decode ladder (2026-10-13): slots are paid in concurrency, not prefill
Frank's challenge — "why are you not tracking decode?" — was right, and the
answer is a number now. 512 generated tokens (the bench law), greedy, cold
prompts, this shape (3 x 200,000, fp8, slab 25,600):
```
1-stream 43.5k: decode 99.0 t/s (prefill 4,042, accept 0.952)
1-stream 197k: decode 93.3 t/s (prefill 4,826, accept 0.938)
3-stream 43.5k: decode 113.9 / 43.3 / 10.9 t/s wall 83.4s
3-stream 197k: decode 88.9 / 16.3 / 64.6 t/s wall 122.4s
```
Two laws came out of it. **Decode is nearly window-free** — 99.0 at 43.5k,
93.3 at 197k, −6 % for 4.5x the depth — so a long conversation costs
prefill, not tokens-per-second. **Concurrency is where the slot count is
paid**: three simultaneous streams give 18.4 t/s aggregate at 43.5k and 12.5
at 197k against 99 single-stream, and the split is a lottery
(113.9/43.3/10.9), not a fair share — the fused-step schedule hands the step
to whoever it reached first, so one stream finishes at 113 t/s while its
sibling crawls at 10.9. That is what "3 slots" owes per token, and no prefill
number shows it.
Corpus caveat, stated at the claim: this probe's filler is a repetitive
ladder, so MTP acceptance is 0.93–0.97 against the 0.740 of the 2026-10-11
table below; these rows flatter decode relative to that table and are the
baseline for *this* corpus, not a replacement for those rows.
`probe-frank-archon-kv.py` now carries this leg as **D** (leg B generates 24
tokens, which is why a day of shape work priced only prefill).
### Recovery evidence
```text
Oct 13 19:21:08 dankserver ttm-preflight.sh[1623778]: frank-archon: ttm pages cap
is 125988 MiB, below the pinned-ledger floor 150000 — raising the knob to
204800 MiB (52428800 pages)
Oct 13 19:21:08 dankserver ttm-preflight.sh[1623778]: frank-archon: ttm pages cap
204800 MiB clears the pinned-ledger floor 150000
Oct 13 19:21:08 dankserver systemd[1]: Started frank-archon.service
```
`cat /sys/module/ttm/parameters/pages_limit` → 52,428,800 (204,800 MiB) after
the boot, i.e. the root `ExecStartPre` hook did the write the service process
(User=dank) cannot.
### Capacity probe (2026-10-13, live service)
`scripts/probe-frank-archon-kv.py` now reads the slot count, window and KV dtype
from the deployed env on the host, so the probe cannot drift from the service:
it printed `shape: 4 x 200,000 tokens, kv fp8` and fired **four concurrent chats
of 196,960 prompt tokens each** (646,299 chars, 98.5 % of the window), each
carrying a marker whose first word is unique.
| Stream | prompt | wall | prefill | retrieval |
|---|---|---|---|---|
| B1 | 196,961 | 238.3 s | 1,423 t/s | exact, HIT |
| B2 | 196,960 | 98.0 s | 2,043 t/s | exact, HIT |
| B3 | 196,961 | 201.1 s | 993 t/s | exact, HIT |
| B4 | 196,960 | 149.9 s | 1,327 t/s | exact, HIT |
| **total** | **787,842** | **238.4 s** | **3,305 t/s aggregate** | 4/4 |
All four answered their own marker; nothing was preempted, refused, or truncated
(`finish=stop`, `cache_n=0`, so no prefix cache padded the result). Aggregate
prefill equals the single-stream rate measured in the same probe (3,397 t/s at
43.5k), i.e. the four streams share one prefill pipeline at the slab-limited
ceiling and the engine never ran short of KV. Prefix cache intact: the same
43.5k prompt repeated answered in 1.0 s with `cache_n=43008`.
The per-stream `decode` column is NOT a decode rate here — 8 generated tokens
make it first-token dominated (the bench law: generate ≥ 128 tokens to read a
rate). This probe answers one question only: does the endpoint owe four
full-height conversations at once, and answer each one exactly. It does.
## Gates (measured at cutover)
- **KLD gate: PASS (2026-10-08 04:12)** — `kld-gate-swift15.sh`,
candidate on 1.0.13 vs `kld-ref`: **mean KL 0.0146** (median 0.00126,
p90 0.0315, p99 0.194, max 4.12), **top-1 97.84 %**,
**ppl ratio 1.0126** (candidate 1.6913 vs reference 1.6702), 50,235
positions / 197 docs. Report: `/home/dank/radiance-state/kld-gate.json`
(+ `.rows` per position). Gate shape: disk-tier, pool 40960, MNBT 3072
(the w4 activation arena scales hard with MNBT: 32768 → 33.3 GiB arena,
refused), KV 2048, headroom 512, no speculation. The engine ran rank 0
fully on card 0 (25.88 GiB) with rank 1 host-side in this shape —
scoring is placement-invariant; scored at 28 positions/s.
- **KLD gate re-PASS (2026-10-08 04:49) at the production-legal shape**:
MNBT 2048 (the w4 file-tier law: experts stream in place only at
MNBT ≤ 2048 — the recording's 32768 demands a widened form the disk
tier refuses), pool 40960, vram-kv 3072 fp8, tp 2, headroom 768.
**mean KL 0.0131, top-1 97.92 %**. The Δ vs the first PASS (0.0015) is
the float-order noise band of the chunk-size change — the verdict is
chunk-size invariant, ~2.3× inside the 0.03 bar on both shapes.
- **Throughput gate: PASS (2026-10-08 05:40)** — decode at 44.5k depth
(streamed chat probe, temp 0, 512 tokens, salted): arm A (main serve
shape, EVC 0.82) **146–173 t/s**, engine timings **121.9 t/s**,
draft accept **0.7246** (350/483); arm C (deployed vram-pair shape
26071/3543) median **136.0 t/s**, warm TTFT 9.45 s. Control
(fp8-native 1.0.13): 30.08 / accept 0.7549. Prefill ~4.7k t/s warm.
The w4 expert table is resident — the fp8-native era's NVMe expert
streaming (the 3.5 GB/s wall) is gone; the MTP ×3 head rides a 45–60
t/s base decode. Conversation battery (short context): geomean
172 t/s (min 100 / max 208), mean ttft 0.09 s. The EVC ladder is moot:
the w4 resident set fits fully inside the 40 GiB pool at 0.82
(ledger: mover full fill). Depth-output audit: the 44.5k-depth
summary is a coherent ordered index of the concatenated corpus
(no repetition, no NaN-token garbage) — the depth-corruption class
of 2026-10-04 is clean.
- Fleet probes: PASS, re-run on the deployed prod — vision (solid-red
512 px → "Red"), tool_calls (`{"expr":"23 * 47"}`), reasoning_effort
xhigh/medium/low/high all 200 (the Swift1.5 froggeric template accepts
the full set), reasoning_content present, response_format json_object
honoured. **Vision probe law:** budget ≥ 256 tokens and read
`reasoning_content` as fallback — at 64 tokens thinking exhausts the
budget and `content` comes back empty (the answer lands in the
reasoning channel).
- **Deployed verification (2026-10-08)**: `deploy.sh dankserver
--service frank-archon` green incl. Playwright e2e 33/33; unit
`active`, `/health` 200, `/v1/models` id `frank-archon` (the harness
alias is the served-model-name, independent of the file — models.yml
untouched). Deployed shape: fp8 KV (KLD-gate shape), spec depth 2,
seqs 3, vram pair 26071/3543. Boot ledger: KV pool 3.46 GiB, expert
pools 57.84 GiB, cards settled 1.41 GiB free, `serving on :7770`.
Live decode probe (512 tokens, streamed, temp 0): ~190 t/s.
### Wire rename + the bench drop-in trap (2026-10-09)
`FRANK_ARCHON_SERVED_NAME` moved the served id `frank-primary` →
**frank-archon** (commit `7c77326`, 2026-10-09); the service, unit, scripts and
env file kept `frank-primary` until the 2026-10-10 rename (`55f87fc`) brought
them to `frank-archon` too. Radiance is the only
fleet engine whose `/v1/models` listing reflects the flag, and it echoes the
**requested** model name in chat responses — so a chat probe that returns 200
proves nothing about the served id. Always read `/v1/models`.
The first deploy after the edit came up green and `/v1/models` still said
`frank-primary`. Cause: a campaign leftover drop-in
`/etc/systemd/system/frank-archon.service.d/bench.conf` (2026-10-08 04:09)
overrode BOTH `Environment=FRANK_ARCHON_ENV_FILE` and `ExecStart` to the
staging worktree `/home/dank/serverAppConfigs-swift15`. Every
`deploy.sh dankserver --service frank-archon` since then restarted the
staging tree: green health gate, old configuration, silently. Verified before
removing: the staging `serve-frank-archon.sh` and `frank-archon.service`
are byte-identical to the checkout, its `frank-archon.env` differed only by
the alias line, and its HEAD (`8375567`) is an ancestor of `main`. The
drop-in was staged in `/home/dank/.AGENTTRASH/20261009-frank-archon-bench-dropin/`
(that dated dir is gone — the trash dir is empty on Dankserver as of 2026-10-18).
After `daemon-reload` + restart, `/v1/models` reports `frank-archon` (boot
healthy in 102 s, shape unchanged: fp8 KV, spec depth 2, seqs 3, vram pair
26071/3543).
**Deployer blind spot:** `deploy.sh` plans from repo paths and gates on the
port. It cannot see host-side drop-ins, so unit drift is invisible to a green
run. After an LLM deploy, verify the served configuration behaviourally
(here: `podman inspect <name> --format {{json .Config.Cmd}}` and `/v1/models`).
## Deploy + verify
```bash
sudo systemctl stop frank-archon # drained window
bash convert-swift15-w4dn8.sh # if not already run
./scripts/deploy.sh dankserver --service frank-archon
```
Post-cutover checklist: unit active/enabled; `/health` 200; boot ledger
(expert pool coverage now w4-sized, KV pool should be larger than the
FP8-native era's 399,824 tokens); MTP draft counts in `timings`; froggeric
probe; ≥256 px `image_url` vision probe answers the color; tool-call
probe; `reasoning_effort` fold; Prometheus target green; `tests/run.sh`
+ Playwright e2e green.
## Rollback
| Layer | Path |
|---|---|
| Publisher iq4r (successor) | [[frank-archon-radiance-qwen38-iq4r]] — the LIVE payload since 2026-10-15 |
| FP8-native radiance | rebuild-only — the `.rad` (173.0 GiB) was `rm`'d 2026-10-15: `scripts/download-frank-archon.sh` + `scripts/build-frank-archon-rad.sh` |
| FreeToken PR #587 | git tag `frank-archon-freetoken-pr587-20261005` |
| tcclaviger vLLM shape G | git tag `frank-archon-vllm-shapeG-20261003` |
**This era's own payload** (`qwen4exp-swift15-w4dn8-i8-hc8m.rad`,
116,416 MiB) was purged in the 2026-10-17 disk cleanup (trash-staged, then
emptied with Frank's approval). The byte-exact container is published as
`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`
(`qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, Hub LFS sha256
`09dd93064365009c508b59012adadab6a36791a50fa319f76b31bf359c01e9ff`) —
re-download restores the exact payload; a full re-convert is no longer
possible without re-fetching a BF16 source (see next paragraph).
Campaign artifacts: the raw BF16 checkpoint was `rm`'d 2026-10-15 (Frank's
one-time authorization), so re-convert needs a fresh BF16 source; `kld-ref`
(24 GiB), grams (`/home/dank/corpora/calib`), corpora JSONL, and the logs
`/home/dank/logs/swift15_*.log` remain. The BF16 `.rad` container was
deleted before the final convert (disk gate).
## Links
- [[frank-archon-radiance-qwen38-native-fp8]] — the replaced FP8-native era
- [[LLM/index]] — fleet index
- [[swift15-campaign-worklog]] — campaign record
- [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here
serverAppConfigs-docs/LLM/frank-archon.md
+419
View File
@@ -0,0 +1,419 @@
---
type: reference
title: "frank-archon on Dankserver — radiance (StillDeadcode) 1.0.13, publisher Qwen3.8-Flash-Next iq4r .rad, 2× R9700 TP=2 (ACTIVE)"
description: "The one frank-archon page: the live radiance shape and harness contract (read off scripts/frank-archon.env), the Eras table carrying every rollback path including the re-download doors for the purged payloads, the laws this endpoint taught (pinned-RAM/TTM preflight, the KV dtype law, the one per-card VRAM budget, descriptor selection), the wire contract, and the files that own it. It replaces the nine per-era runbooks."
tags:
- llm
- frank-archon
- dankserver
- radiance
- gfx1201
- r9700
- tp2
- expert-offload
timestamp: 2026-10-18
---
# frank-archon — radiance (StillDeadcode), 2× R9700 TP=2, Dankserver :7770
The radiance engine (C++/HIP, `libr4d` kernels built for gfx1201, no Python in the serving
path) driving `podman run` straight from the digest-pinned image under a **root** systemd
unit; container name = unit name. Host Dankserver (192.168.0.235), port 7770, both Radeon AI
PRO R9700s. Capacity contract **3 conversations × 200,000 tokens at fp8 KV**; a payload swap
is a `MODEL_FILE` line plus a re-download — see Eras.
## Current shape
Authoritative source: `scripts/frank-archon.env` + `scripts/serve-frank-archon.sh` (the env
header carries the measured arithmetic beside each number).
**Payload** — `/home/dank/models/rad/qwen3.8-next-flash-fp8-iq4r-moe.rad`, 121,969,901,568 B
= **116,322 MiB**, sha256 `0af5e962…` (the Hub LFS oid), from
`StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe`: the engine author's own rad-convert of the
**official (censored)** `Qwen/Qwen3.8-Flash-Next` bf16 checkpoint. Encodings (recipe
`qwen4exp-w4nl64-i8-hc8m`): routed experts as 4-bit codes in a 16-level non-uniform codebook
(128-pt Walsh–Hadamard rotation, GPTQ on a 10M-token calibration + 3 coordinate-descent
sweeps, E4M3 scale per 64 weights); **10 whole bf16-protected experts per layer**; int8 W8A8
trunk + lm_head; fp8 HC mixtures; PLE n-gram fp8 at the checkpoint's fixed scale (47.7 GiB);
MTP head and the bf16 27-layer vision tower **in the container**. Card quality vs bf16: mean
KL 0.0789 all-tokens / 0.0239 assistant turns, top-1 92.85 %. The abliterated family lives on
[[frank-dragoon]]. Fetched by `scripts/download-frank-archon-iq4r.sh --fetch` (hfget resume +
declared size + sha256) as the transient unit `iq4r-fetch` (log
`/home/dank/logs/iq4r-fetch.log`, `…rc` sentinel); the staging dir
`/home/dank/models/StillDeadcode/…` keeps card, LICENSE and recipe as provenance, and only
`.rad` files are mounted.
**Runtime** — `stilldeadcode/radiance@sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e`
= pin **1.0.13** (Codeberg `v1.0.13`, commit `53282fa`): a 287 MiB image in dank's
**rootless** storage, host needs nothing but the amdgpu driver. Pin the INDEX manifest digest,
never the image ID (`69d05724…` is the config digest and sends `podman run` to the network).
The serve script's gate is that digest ref itself — `podman image exists name@sha256:…` in one
command (the 2026-09-30 no-pipe/grep-race lesson) — so a retagged look-alike or a newer
`latest` cannot satisfy it. 1.2.0 is a rejected serve pin (Laws); the
`--vram-weights-mib`/`--vram-kv-mib` pair is accepted by both pins, which is the rollback symmetry.
**Trio** — `scripts/frank-archon.env` (every number, with its arithmetic in the header),
`scripts/serve-frank-archon.sh`, `scripts/frank-archon.service`. Gates run in order: image
digest present → `.rad` present and ≥ `RAD_MIN_MIB` (a truncated convert would otherwise buy a
guaranteed long-timeout boot instead of this fast failure) → template present → `/dev/kfd` +
`/dev/dri` → `ttm_ensure_cap` (the serve script re-verifies the cap after the unit's hook) →
`podman rm -f` over `FRANK_ARCHON_LEGACY_CONTAINERS` (`frank-archon-qwen frank-archon-r9v
qwen-fp8 glm-probe`) plus its own name, because systemd SIGKILLs conmon on a crash and leaves
the name registered, and every one of those identities holds this GPU. Container contract
(publisher-vetted, `docs/DOCKER.md`): `--network host`, `--init` (so the engine releases the
cards on SIGTERM), `--security-opt seccomp=unconfined` (the default profile refuses io_uring —
the PLE table's reads — and the NUMA calls the ROCm runtime makes for host memory), `--ulimit
memlock=-1:-1`, `/models` ro + `/data` rw + the tune cache at `/opt/radiance/share/radiance/tune`,
template as a single-file mount, then `exec podman wait`: the script's lifetime IS the
container's, so a crash is a nonzero `ExecStart` and the restart policy is real.
Unit: `After=network-online.target`, `WantedBy=multi-user.target`, `User=dank`,
`LimitMEMLOCK=infinity`, `StartLimitIntervalSec=0`, `OOMScoreAdjust=-500` (the last process the
OOM killer should pick), `Restart=always` **not** `on-failure` — a clean API-server exit after a
worker fault reads as success to systemd (the 2026-10-01 vLLM lesson) — and
`TimeoutStopSec=180` > the `ExecStop=podman stop -t 120` grace, because systemd's 90 s default
cut the stop short and recorded `Failed with result timeout` (measured 2026-10-01 13:27:17). No
drain contract is claimed: the OpenAI front end holds no queue, so an in-flight decode is lost
on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_period`).
| Knob (env name, `FRANK_ARCHON_` dropped) | Live value |
|---|---|
| TP / wire | `TP=2`, `TP_WIRE=wht6` — libr4d one-shot all-reduce, not RCCL; `min-kb` stays the engine default |
| Slots / window | `MAX_SEQS=3`, `MAX_MODEL_LEN=200000` (native 262,144; longer is refused, never truncated) |
| KV dtype | `KV_DTYPE=fp8` — the only dtype that funds 3 slots (Laws) |
| VRAM pair | `VRAM_WEIGHTS_MIB=25600` + `VRAM_KV_MIB=5120` = 30,720 of the 30,969 ceiling; `GPU_HEADROOM_MIB=96` |
| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=44928` **per rank** |
| PLE | `NGRAM_PLACEMENT=ram` (47.7 GiB pinned; refuses boot on a short memlock) |
| Spec / batch | `SPEC_TOKENS=2` — the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); the publisher's card ships depth 3 as its peak and this fleet's decode ladder was measured at 2. `MAX_BATCHED_TOKENS=2048` |
| Prefix cache | `PREFIX_HOST_MIB=8192` + `PREFIX_DISK_MIB=65536` under `STATE_DIR=/home/dank/radiance-state` (mounted rw at `/data`) |
| Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it |
| TTM gate | `TTM_MIN_MIB=150000`, `TTM_TARGET_MIB=204800` |
| Boot | `RAD_MIN_MIB=110000` deployer floor; the listener binds only after the full load (health ~100 s, budget 1800 s) |
Boot ledger for this shape (2026-10-15, pin 1.0.13): pool 5.00 GiB backs **607,740 tokens**
(`kv_attn` fp8 page 26.00 KiB, `kv_qsa_bkey` 3.25 KiB), `kv_gdn_state` 54 MiB × 6 slots, slab
**17,548 slots** with 4.22 GiB of KV lent to it, sampler **9 rows**, template 28,234 B, vision
live (≤ 4,194,304 px, 16,384 patches/pass).
The boot **ledger**, not this arithmetic, is the referee: radiance states every budget before
anything is allocated (weights vs KV, the host pool's coverage of the worst case, the PLE pin,
the P2P answer per pair) — read it after every resize, and unlike every predecessor there is no
per-boot VRAM planner to second-guess. For comparison, the card's own suggested run line on 32
GB cards is `--host-pool-mib 12288 --expert-vs-cache-ratio 0.82` with seqs 8 and spec 3; the
shape above is the fleet's measured one. `FRANK_ARCHON_EXTRA_ARGS` is the boot-ladder escape
hatch (unset → no extra argv; it is expanded unquoted on purpose, so keep multi-flag values as
one string).
**Harness contract** (`/home/blank/.omp/agent/models.yml`) — provider `llamacpp (dankserver)`
→ `http://192.168.0.235:7770/v1`, `api: openai-completions`, `auth: none`, `id: frank-archon`
(= `FRANK_ARCHON_SERVED_NAME`; the provider *label* is legacy and renaming it breaks
`config.yml` references). `contextWindow: 200000` must equal `MAX_MODEL_LEN`;
`maxTokens: 16384` (clamp `min(maxTokens, window − estimate − 4096)`; compaction fires at
85 % → 170,000 est × 1.03 + 16,384 ≤ 200,000); `input: [text, image]`;
`compat.reasoningContentField: reasoning_content`; `reasoningEffortMap` minimal/low→low,
medium→medium, high/max→xhigh; `extraBody` temperature 1.0 / top_p 0.95 / top_k 20 / min_p 0 /
presence_penalty 0 / repetition_penalty 1.0 + `chat_template_kwargs {enable_thinking,
preserve_thinking}`, every key inside the engine's known-parameter list (unknowns 400).
`compactionModel` routes to `llamacpp (dankclaw-3090)/frank-zealot`.
**Deploy / verify** — `./scripts/deploy.sh dankserver --service frank-archon`; launch it
detached when the swap restarts the endpoint your own session runs on. Then verify
**behaviourally**, not by the green gate: `podman inspect frank-archon --format '{{json
.Config.Cmd}}'` and `/v1/models` — `deploy.sh` plans from repo paths and gates on the port, so
host-side unit drift is invisible to it.
## Eras
One row per retired era on this port. The **Rollback** cell is the load-bearing one: anything
marked "re-fetch" is gone from `/home/dank/models/`, which today holds only `StillDeadcode`,
`rad` and `tpl.jinja`.
| Window | Payload | Runtime | Shape | Replaced because | Rollback |
|---|---|---|---|---|---|
| 2026-10-08 → 10-15 | in-house `qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB (Swift1.5 abliterated BF16, GPTQ w4 experts + 24 bf16 protected, KLD 0.0131 / top-1 97.92 %) | radiance 1.0.13, same digest | 3 × 200,000 fp8 KV, slab 25,600 / pool 5,120 — the shape the live file still carries | the engine author's own container is the supported path and his quality ceiling; the finetune's refusal-bending left the endpoint | `.rad` purged 2026-10-17 (trash-staged then emptied, Frank's approval). Re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` → `qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`, via the iq4r fetch script's hfget/verify pattern retargeted at that repo; point `MODEL_FILE` at it, redeploy. Re-**convert** is dead: the BF16 source was `rm`'d 2026-10-15 (`kld-ref`, grams, corpora and `/home/dank/logs/swift15_*.log` survive) |
| 2026-10-05 → 10-08 | `qwen4exp-fp8-native.rad`, 177,109 MiB (173.0 GiB) — native modelopt block-FP8, no requantisation | radiance 1.0.7 line, same digest | TP=2 wht6, 200,000 × **2** slots (3 later), slab 26,071 + fp8 KV pool 3,543, host pool 40,960/rank, `--expert-vs-cache-ratio 0.82`, spec 3, MNBT 2048; that pair measured 30.08 t/s decode on the 2026-10-06 43.6k greedy probe and 4,379 t/s A-cold prefill on 2026-10-11 | native FP8 experts (56.4 GiB/rank) ate a 40 GiB pool and left KV at 3,543 MiB = 386,904 tokens, **1.93 sessions** — context economics, not quality; and the host pool could not be trimmed to pay for it (36,864 refused the boot) | rebuild-only: `scripts/download-frank-archon.sh` (official `Qwen/Qwen3.8-Flash-Next-FP8`, 131 shards, 172.8 GB, ungated) + `scripts/build-frank-archon-rad.sh` (no-recipe convert under the descriptor law); raw **and** `.rad` were `rm`'d 2026-10-15 (Frank's call, to free the volume for the iq4r download) |
| 2026-10-05 (same day) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` safetensors, 131 shards | FreeToken PR #587 head `77a7201` (`rocm714_torch214`), image `localhost/frank-primary-freetoken:pr587-77a7201` | ONE R9700; KV sized in tokens `299,968` (a multiple of 64 — the `qsa_sparse` backend pins page 64), 2 running requests, `--moe-cache-rate 0.10`; 17.1 t/s decode | TP=2 cannot boot on ROCm: the per-rank `pynccl.cu` JIT cannot resolve `cuda_runtime.h` for gfx1201, so the second card idled | git tag `frank-archon-freetoken-pr587-20261005` (trio + `scripts/docker/frank-primary-freetoken.Dockerfile`, which exists only there); image still in rootless storage; payload needs a re-fetch, and that repo 403s until the gate is accepted on the account |
| 2026-09-29 → 10-05 (interrupted 09-30 → 10-01 by GLM) | same orcarouter FP8 safetensors | tcclaviger vLLM fork, `docker.io/tcclaviger/vllm@sha256:da402966…` | shape G: 3 × 200,704 fp8, KV 3,166,500,000 B/rank → 401,408 tokens, arena 17.77 GiB / 7,762 slots, util 0.982, MNBT 2048, no spec, no prefetch, prefix caching ON | radiance gave real TP=2 (no RCCL) and a 2-3 min boot, and the fork never fixed its long-context class | git tag `frank-archon-vllm-shapeG-20261003` (trio + the 362-line env lore); image still in dank's rootless storage; payload needs a re-fetch. The 2026-10-01 fp8+MTP4 rung is tag `frank-archon-fp8-mtp-20261001` |
| 2026-09-30 → 10-01 | `Intel/GLM-5.3-Flash-W4A16-AutoRound` ~170 GiB (sym int4 g128, `auto_gptq`) | same tcclaviger digest; `R4dW4a16MoEMethod` → libr4d grouped W4A16 | TP=2 + `--enable-expert-offload --expert-precise-memory`, 2 × 131,072 bf16 KV, util 0.98, MNBT 2048, 2,383 slots + 73.64 GiB pinned per rank; 9.9-10.0 t/s decode / 918 t/s prefill | the 35 t/s target is unreachable as the fork ships: 44 % of the ~93 ms/token budget is pointwise glue in an **uncompiled** GLM trunk — the expert gather is 1.05 % of device time | payload (170 GiB) + drafter (2.2 GiB) went to `~/.AGENTTRASH/20261001-glm-purge/` and were emptied; the cutover was reversed by `d91379c`, restoring the `2339fae` trio; `scripts/patches/glm53-*.patch` survive in-tree but none applies byte-exact since the 2026-10-10 rename rewrote their context lines |
| 2026-09-27 → 09-30 | unsloth `Qwen3.8-Flash-Next-GGUF` Q8_0, 175.30 GiB / 6 shards + blockfp8 MTP4 head + mmproj; PLE derived 54,400,261,120 B | R9V radiance-vLLM fork `drwolfen/radiance-vllm-qwen4exp`, docker bundle `2dac17a2`, behind the docker→podman shim | `MAX_MODEL_LEN=131072`, 80-slot tiered cache (the policy max), MTP4, PLE **pinned**, prefix caching ON from 09-28 | the tcclaviger fork took the port for the GLM W4A16 experiment; Q8_0 had halved expert residency and a 128K envelope | host state survives: `/home/dank/r9v-state/q8xl/setup.json`, the fork checkout `/home/dank/GitPackages/radiance-vllm-qwen4exp` (local-only commits `363d606`/`30d4a32`/`2de0357`) and `.podman-shim/docker`; payload re-fetch first |
| 2026-09-27 (same day) | `Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS`, 92.39 GiB + PLE 26.82 GiB (`34fa36f8…`) | same R9V bundle + shim | `MAX_MODEL_LEN=196608`, offload budgets 146.25 / 78.75 GiB, PLE residency `ssd` mmap, tool parser `qwen3_coder`, thinking on the `reasoning` (vLLM) channel | same-day flip back to the reference quant with the MTP4 head, at Frank's direction | `/home/dank/r9v-state/uncensored/` (`setup.json`, `restore-config-20260927.env`) is still on disk; the IQ4_XS payload was deleted in the 2026-10-17 cleanup → re-fetch, point `R9V_STATE_FILE` back, drop the two q8-only keys and the template keys |
| 2026-09-26 → 09-27 | unsloth Q8_0, 175.30 GiB / 6 parts + shared-Q8_0 MTP 2,786,568,256 B + `mmproj-BF16.gguf` 907,542,944 B | unsloth llama.cpp b11030-mix prebuilt (commit `6ba30d05b`), ROCm gfx120X | `-ts 1,1` layer split, routed experts **and** the PLE table computed in host RAM (`-ot exps=CPU,per_layer_token_embd.weight=CPU`), 262,144 × 1 slot, then the same-day reshape to 4 × 262,144 with V `q8_0` and hot expert blocks 5/3 | the R9V radiance-vLLM fork took the port at Frank's direction (true TP=2 plus MTP4) | trio at commit `4992e5a`+; the GGUF/MTP/mmproj set is no longer on disk → re-fetch, then redeploy |
| 2026-09-26 (same day) | `Lucebox/DeepSeek-V4.1-Flash-ROCMFP23-GGUF`, 377.2 GiB, sha256 `2daea6ca…` | `localhost/lucebox-rocm:gfx1201`, built from `Dockerfile.rocm` + 3 local patches | ONE R9700, `--max-ctx 131072`, `--chunk 8192`, q4_0 K+V, `--ds4-prefill sparse`, routed experts on CPU from the page cache | 377 GiB fits neither 63.7 GiB of VRAM nor 246 GiB of RAM → NVMe page-cache thrash; Qwen Flash-Next Q8 gave better quality, vision and 262k at comparable speed | image still in dank's rootless storage; payload gone; the quadlet and its entrypoint were deleted by `8500600` — `git show 8500600^:dankserver/frank-primary/quadlet/frank-primary.container` (and `…:dankserver/frank-primary/frank-primary-entrypoint.sh`), stop the root unit, start the user quadlet |
Before 2026-09-26 the port belonged to the **`frank-medium`** service: [[frank-medium-radiance-mxfp4]]
(vllm-radiance TP=2 MXFP4, 2026-09-20 → 09-26) is that era's page and is **not** superseded here.
## Laws learned here
### The radiance engine
- **KV dtype law.** `--kv-cache-dtype` takes only `bf16|fp8` — ONE dtype for the whole cache,
so llama.cpp's exact-K / quantized-V pair (what [[frank-zealot]] runs as `--cache-type-k`/
`--cache-type-v`) is unservable here: the rung below bf16 is fp8 on both halves. Measured
per-rank cost out of the boot ledger's group table (fp8 boot 2026-10-08, bf16 boot
2026-10-11): `kv_attn` 13 layers / 4-token blocks pages **26.00 → 52.00 KiB**, while
`kv_qsa_bkey` stays **3.25 KiB at both** — the QSA block key is an index, not attention state,
and does not follow the flag. So 7.3125 KiB/token at fp8 vs 13.8125 at bf16: **1.89×, not
2×**, precisely because the bkey plane is dtype-fixed. Secondary saving: a prefix-cache entry
tracks the attention plane (110.50 KiB at bf16 → 58.50 at fp8, per the `kv tier transfer`
lines), so the same 8 GiB host tier parks ~5.8 sessions of 200k at fp8 where it parked ~2.9 at bf16.
- **Pool sizing law.** `pool_mib ≥ tokens × 7.3125 KiB + 110 × seqs + 448` — the dtype-
independent terms are also in the pool: `kv_gdn_state` 54 MiB × 2 copies per sequence (+conv/
tail/ple ≈ 110 MiB a sequence) and the prefix checkpoint region 8 × 55.99 MiB. Deployed: 3 ×
200,000 at fp8 = 4,285 + 330 + 448 = **5,063 MiB**, pool 5,120, which predicts (5120 − 330 −
448) × 1024 / 7.3125 = **608,027 tokens** against a 600,000 contract (1.3 % slack) — every
MiB above 5,063 was handed to the expert slab instead. At bf16 the same three slots cost
8,093 + 778 = 8,871 and the slab falls to 20,249 (−9 %); four slots at bf16 cost 10,791 + 888
= 11,679 and the slab falls to 17,441 (−33 %); fp8 keeps 22,208 of slab at both slot counts.
**fp8 is not a preference, it is what funds the slot count.** Measured backings to sanity-
check the formula: bf16 pool 6,912 at seqs 2 backed 462,752 tokens (2026-10-11), the fp8-era
pool of 3,543 gave 386,904 (1.93 sessions), and the four-slot fp8 boot of 2026-10-13 19:21
backed 843,276. The ledger's `backs N sequences all at 200000 tokens at once` line is the
referee — a pair this tight is read off the boot, never assumed, and the concurrent probe is
its regression test. Fourth full-height conversation needs 6,601 MiB of pool: refusing it is
Frank's explicit call, and serving it is a `--vram-kv-mib` change, not a code change.
- **The hard limit is per request.** The engine treats `kv group tokens >= max-model-len` as a
HARD limit — "one request longer than N tokens is accepted by the API and then refused as too
large to ever fit — this is a hard limit, not a backing ratio" — so the pool is sized for
`seqs × max-model-len`, not for one sequence. Radiance never truncates, at any size.
- **One VRAM budget per card:** `weights + kv ≤ 30,969 MiB`, read off the engine's own budget
table (boot 2026-10-13 at pair 24000/5120): card 32,624 MiB, budgets claimed 29,120, and
"*3.42 GiB is neither budgeted nor accounted — the driver's context, the activation arena and
`--gpu-headroom-mib` come out of it; anything past those is throughput on the floor*", with
the card settling 2.01 GiB free against a 209.19 MiB floor (96 headroom + 113.19 kept for the
static region). Reserve above the engine's floor = 2,058 − 209 = 1,849, ceiling 29,120 + 1,849
= 30,969; the deployed pair lands 458 MiB above that floor. The 2026-10-11 bound of 29,120
("31,856 − 1,710 outside both budgets − keep 1 GiB free") was a hand-picked reserve, not the
engine's wall. Boot with the explicit `--vram-weights-mib`/`--vram-kv-mib` pair, never
`--kv-cache-size-mib`; with the disk tier engaged the engine also forces ~9.6 GiB of weight
buffers + arena onto the weights side (~29.6 GiB/card effective).
- **Idle KV VRAM is the expert cache's spillover — the price is paid twice.** `mover: 2275
slots can be lent out of the KV cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <=
256 tokens`; the lend grows with the pool (2.69 GiB from 3.46, 6.08 from 6.912, 4.22 today),
so a bigger KV pool steals weight residency from the slab AND from the lendable cache. Priors
on this card/model: stealing 2.14 GiB of weights to buy `MAX_MODEL_LEN 262144` cost 2.5 %
decode; the other direction (`--expert-vs-cache-ratio` 0.82 → 0.84, +2.5 GiB of weights) cost
8.9 %.
- **The host pool moves with the slab.** `vram_weights` and `host_pool_mib` serve the per-rank
resident set *together*: at the fp8-KV pair (26071/3543) the ledger showed the slab full to
259 KiB of slack, and cutting `vram-weights` by 3.8 GiB to pay for a bf16 pool spills ~3.8
GiB/rank into host RAM — under-sizing the pool there is a **boot failure** (36,864 refused the
boot in the fp8 era). The ledger's coverage line, not the arithmetic, arbitrates.
- **Slab → prefill: the arms disagree, the cliff does not.** 2026-10-11 exact-KV A/B (slab
22,208 vs the fp8-era 26,071, −14.8 %): prefill −29 %, TTFT at the full window +25 %, decode
inside the noise band on 2 of 3 passes, draft acceptance UP 0.661 → 0.740. 2026-10-13
(`probe-frank-archon-kv.py`, env header): A-cold 43.5k 22,208 → 3,397, 24,000 → 3,290, 25,600
→ **3,584** t/s, and the 3-way 197k arm's aggregate 3,051 → 3,589 (wall 193.7 → 164.5 s), so
+1,600 MiB of slab bought +8.9 % single-stream / +17.6 % concurrent — read then as a SLOPE,
the 22,208/24,000 pair having landed inside noise of each other and falsely looked like a
plateau. Re-measured at 3 slots after the 2026-10-15 swap the 25,600 arm returned 3,290: the
3,584 did **not** reproduce, so treat 22,208-25,600 as one band until someone re-runs the
ladder. What is not disputed is the cliff: 26,071 → 4,379 t/s (the 2026-10-11 control at seqs
3 / pool 3,543), +22 % for +471 MiB — unreachable at 3 slots (26,112 + 5,120 = 31,232 vs
30,969) and free at 2 (pool 3,072 + slab 27,897 = the ceiling exactly). Two points inside
noise of each other are not a plateau, and two points are not a curve.
- **Decode is nearly window-free; slots are paid in concurrency.** 99.0 t/s at 43.5k vs 93.3 at
197k (−6 % for 4.5× depth), while three streams give 113.9 / 43.3 / 10.9 = 18.4 t/s aggregate
— a fused-step lottery, not a fair share. Read it from leg D of `probe-frank-archon-kv.py`
with ≥ 128 generated tokens; an 8-token probe measures first-token noise.
- **MNBT 2048.** 2560 collapses decode to 2.6-7.7 t/s (MTP resonance); the w4 file tier streams
in place only at MNBT ≤ 2048 and the activation arena scales with the chunk (32768 → 33.3
GiB, refused).
- **Slots cost KV and rows, never speed.** `--max-num-seqs` carries no radiance constraint: it
adds 110 MiB of recurrent state per slot and declares the fused TP all-reduce over `seqs ×
(1 + spec)` rows, and `wht6` is bit-exact up to 25 rows (single-sequence decode and any step
of ≤ 25 rows is exact; only busier steps are lossy) — 3 × 3 = 9 today. Measured 2026-10-06:
seqs 2 and seqs 3 are statistically tied (30.9-33.1 t/s, inside the ±3 % floor). The "seqs 2
cannot boot" law belonged to the retired tcclaviger fork's `torch.compile` shape
specialization, and radiance booted seqs 2 on its first fp8-native start.
- **One encoding per expert table** (`rad_arch_declare`): a whole expert kept bf16 is the only
thing legal beside the table, so a split tier (down=fp8, gate_up=w4) is unservable.
- **Descriptor-selection law.** The plugin is picked by the exact `(arch_id, quant descriptor)`
pair with no fallback: a raw official FP8 checkpoint (`quant_method: "fp8"`) is refused by
serve *and* convert, because the shipped `qwen4exp_fp8` plugin claims `(qwen4exp, "")`. Stage
a `config.json` with `quantization_config` dropped — the descriptor changes, the data stays FP8.
- **Three convert laws, each paid in a full verify+declare cycle.** Scale-name: `scale_name()`
appends `_scale_inv` to the *tensor* name, so stacked `…experts.gate_up_proj` keeps its scales
at `…gate_up_proj_scale_inv`. File-shadow: the reader opens index-named files sorted and
emplaces every tensor each file physically holds, first-wins — a merged PLE shard must sort
first (`00-ple.merged.safetensors`) or a raw shard wins the name. Symlink-through: staging
symlinks the raw index, so `open(path,"w")` writes through it and corrupts the checkpoint —
write temp + `os.replace`.
- **Convert is card-less** (`rad-convert` declares on CPU, GPTQ solves on CPU BLAS) and
`--plan-only` goes first: a plan that resolves every op will boot. `rad-tune` must be
declared at the shapes you serve and its cache lives in the *image* (`$RADIANCE_HOME/tune`),
so it only survives on the `$FRANK_ARCHON_STATE_DIR/tune` bind-mount the serve script adds.
- **Quant gates are scored at the production shape** (MNBT 2048, not the recording's 32768)
against the BF16 container: KL ≤ 0.03, top-1 ≥ 95 %, and the verdict is chunk-size invariant
(arm Δ ≈ the 0.0015 float-order band) and placement-invariant. Protected experts are
data-driven — energy share ≥ 3 % AND ≥ 3× routing share, even count per rank half — and the
PLE rides the checkpoint's own fixed scale (1.9931793212890625e-04, absmax inside the
2⁻¹³..2⁻¹⁰ plateau).
### This host, and the deploy path
- **The pinned-RAM ledger is why this is a root unit:** 2 × 44 GiB expert pools + the 47.7 GiB
PLE n-gram table (51B params, read from pinned host RAM over io_uring — Frank's pin law:
"*Native FP8. Experts from ram. Pin the PLE to ram.*") + the 8 GiB prefix host tier ≈ **147
GiB** charged against `RLIMIT_MEMLOCK`; dank's login session caps at 8 MiB and a user quadlet
is capped by the user manager, so only PID1 can grant `infinity` — and
`--ngram-placement ram` refuses the start when the limit is short.
- **TTM is runtime state, and the gate that owns it raises before it refuses.** The pinned pages
ride `/sys/module/ttm/parameters/pages_limit` (+ `pages_max` on kernels that expose one; this
kernel exposes **none** — the cap is the MIN over the knobs present, and an absent knob is
skipped, never read as zero, because a zero would refuse a healthy cap). The kernel default is
**half of RAM**, so every raise dies with the reboot: the 2026-10-05 manual raise to
52,428,800 pages (200 GiB) was gone after the **2026-10-12 23:17 power failure**, the cap fell
to 125,988 MiB, and the preflight refused the start **6,919 times over ~20 h** while
`Restart=always` cheerfully re-ran a refusal. A manual sysfs write is therefore not an
acceptable repair procedure — `scripts/ttm-preflight.sh` writes `TTM_TARGET_MIB` (204,800 MiB
= 52,428,800 pages; pages are 4 KiB so pages = MiB × 256; the *recorded host ceiling*, because
writing the 150,000 floor would leave the ledger no slack) into every knob present, re-reads,
and only then refuses. Write failures are swallowed deliberately: the caller re-reads, and ttm
itself refuses a limit below the pages it has already pinned.
- **Privilege of that knob, measured the hard way:** it is root-owned 0644 and the service runs
`User=dank`, where `-w` is false and a plain `echo` fails. The canonical privileged path is the
unit's `ExecStartPre=+…` (the `+` runs it as root even under `User=`); `sudo -n` is the
fallback for a human. The file is *sourced* by the serve script (which re-verifies the cap
before `podman run`) and *executed* by the unit. A refusal is invisible outside the journal —
no container is created, the unit sits in `activating (auto-restart)` on one repeating line —
and without the gate the load instead dies mid-pin with a bare allocator error.
- **The prefix host tier needs real RAM:** 8 GiB of it, and with the service loaded the host
still has ~104 GiB available (246 GiB total) — that headroom is what the pinned ledger is
measured against.
- **The deployer's blind spot is host-side unit drift.** A leftover
`frank-archon.service.d/bench.conf` overrode both `FRANK_ARCHON_ENV_FILE` and `ExecStart` to a
staging worktree and every `deploy.sh` since was green against the old configuration —
drop-ins are read *after* the unit's own `Environment=` lines, so they win.
- **Never `systemctl disable` a unit on these hosts** — it removes the `/etc/systemd/system/`
entry, which here is the symlink into the checkout. `mask` is reversible but *fails on a
relinked unit*, so a bench window uses `stop` and the deploy restores the unit.
- **The restore oracle for an experiment window is a conjunction:** `is-active` == active AND
`is-enabled` == enabled AND no `/run/systemd/system/<unit>` mask symlink AND the container is
owned by the unit and not `Stopping`. The vLLM/GLM-era engines ignored SIGTERM and answered
`/health` 200 for 20+ minutes while `Stopping`; a runtime mask survives `stop` and blocks
`start`, and a failed unit needs `reset-failed` first.
- **`MemAvailable` is an input, not a backdrop:** the expert-offload planner reads it, so
anything booted beside the live service dies at plan time (`ExpertPlanError: host holds 5113
of 12096 swappable experts`) — and fleet law forbids GPU experiments beside production
anyway. Long remote work runs as a transient unit or `setsid`+disown: `nohup X &` inside `ssh`
races ssh teardown and can die before writing a byte.
- **The ~40k-token corruption class was the RAM, not the engine.** Dropping DDR5 from 4800 to
4600 MT/s ended it, with zero `gdn_hip` faults either side; the prefix-caching / GDN re-seed
theory was confounded on both arms. Consumer non-ECC raises no MCE for a silent flip and a
60 s / 8 GiB pattern check covers 0.5 % of 246 GiB — silent RAM corruption outranks every
software theory.
- **TP=2 here** is legal since the 2026-09-20 ProArt B850 swap (x8/x8 on CPU root ports,
`Routing+`/`ReqEn+` end to end) only because radiance's wire is `libr4d`'s own one-shot
all-reduce; RCCL needs atomics on the whole chain and fails `hipErrorIllegalState` without
them ([[Dual-R9700-Cooperation]]). `wht6` buys +5.6-6.3 % prefill for ~1.3 % of 8-stream
decode, `exact` is the bit-exact alternative.
- `--prefix-cache-dir` creation is **not recursive** — the serve script `mkdir -p`s the path or
the disk tier silently never opens. The tier does survive a payload swap when arch and
tokenizer match (it recovered 900,176 blocks on the 2026-10-15 swap; `cache_n` proves a hit).
### Dead ends — do not re-try
- **radiance 1.2.0 as the serve pin** (`sha256:665775ed1d90de6dbb25558e07fc60d20328f1346f96f3c6e00845eb763f01fa`,
commit `e95d397`), benched 2026-10-06 on this exact shape with a 43.6k greedy probe: decode
29.21 t/s and draft acceptance 0.7123 vs the 1.0.13 control's 30.08 / 0.7549, and the control
reproduced the recorded 0.755 exactly — greedy acceptance is deterministic, so the drop is a
real numerics change (the decode loss tracks the acceptance ratio, 0.943 vs 0.945) with the
1.2.0 MoE weight-rotation change `92b8aaf` the named suspect. The vram-pair translation itself
is exact (the 1.2.0 ledger took 25.46 GiB weights / 3.46 GiB KV per card, byte-identical to
1.0.13 at EVC 0.82), and 1.2.0 REMOVED `--expert-vs-cache-ratio` in favour of the pair, which
both pins accept. Re-test 1.2.0+ once the rotation change is untangled.
- **FreeToken TP=2 on ROCm:** the `pynccl.cu` JIT cannot resolve `cuda_runtime.h` for gfx1201 —
an untested edge of that PR, not a board fault.
- **tcclaviger forks:** `--max-num-seqs 2` (`ConstraintViolationError` on `query_start_loc`);
`--gpu-memory-utilization 1.00` unbootable, 0.982 is the law; expert prefetch is poison (36 %
precision — deleting it was +40-51 % decode); TunableOp-with-offload corrupts numerics
(repetition loops, early EOS) — autotuning that changes text is broken tuning; ngram-only
drafting dies at init (`PLE inputs were not prepared`); DCP needs TP > `num_kv_heads` (2 here,
so 4 GPUs); 6 × 262,144 fits no KV dtype (6 sequences leave 9.18 GiB of pool where fp8 needs
12.47). fp8 KV on that path also removes the only MLA decode gfx12 has (the aiter op is gated
on CDNA > 2, so the draft step demands `…mla_decode_fwd`, which cannot exist there) and its
KV reserve model is dtype-blind (identical 13,600 B/token plan line) — no capacity for the breakage.
- **DeepSeek-V4.1-Flash on 32 GiB cards:** only `--ds4-prefill exact` is legal,
`--ds4-fused-decode` is refused, and at ~22 % resident experts each token pulls ~187 × 12 MiB
from NVMe — low GPU util with a pegged link is a *capacity verdict*. Two corrections survive:
expert TP **does** work on gfx1201+gfx1201, and an expert-placement JSON outside the mounted
`/models` tree gives a silent single-tier fallback.
- **Which experts you pin does not matter; how many does.** An IBS census of the live server
(`scripts/frank-archon-expert-activity.py`) measured per-block read volume flat at 0.93-1.08×
of mean across every CPU-resident block — top-k routing fires all of them every token, so at
`-ot` granularity the lever is COUNT.
- **Never mount a diagnostic hook on `PYTHONPATH` without a marker guard:** `sitecustomize` is
imported by every interpreter, a child inherits the hook and spawns again — a 19-level python
fork bomb inside the container. Kill with `podman rm -f` (the cgroup owns every generation),
never pattern-kill an interpreter name on this host; use `TORCH_LOGS` for dynamo questions.
## Contract (the wire)
- **Live surface:** `/health` (== `/ping`), `/v1/models`, `/v1/chat/completions`,
`/v1/completions`, `/metrics`, `/stats`, `/sessions`, `/dashboard`. `/health` answers 200
JSON (model id, `in_flight`, `queue_depth`) while serving and **503 only while stopping**
(`core/server/admin.cpp:385`), and the listener binds *after* the full load (`main.cpp`:
`engine.start()` then `engine.serve()`) — during the load the port refuses connections, so a
bare `curl -fsS …/health` gate is honest and needs no body grep.
- **Served id.** `FRANK_ARCHON_SERVED_NAME=frank-archon` (the wire rename of 2026-10-09 moved
only the alias; container, unit, scripts and env keep the service name). The env's contract
line is that **radiance validates the requested model name, so every client — harness, bench
scripts, probes — MUST send it** — while the era pages also record that a chat response
*echoes* the requested name, which is why `/v1/models` is the served-id oracle and a 200 chat
probe proves nothing about what is actually loaded.
- **Reasoning** lands in `reasoning_content` (`--reasoning-format` defaults to `auto`; no
parser flags exist or are needed) and `usage` bills it separately (`reasoning_tokens`).
**Thinking consumes `max_tokens`**, so the classic false negative is a probe with empty
`content` and `finish_reason: length`/`stop` because the whole budget went inside the think
block: either set `chat_template_kwargs.enable_thinking=false` or budget a few hundred tokens
and read `reasoning_content`. Same trap, sharper, on the vision probe below.
- **Sampler surface:** `temperature, top_p, top_k, min_p, typical_p, presence_penalty,
frequency_penalty, repetition_penalty, repeat_last_n, chat_template_kwargs, reasoning_effort,
enable_thinking, preserve_thinking` accepted; **unknown parameters are refused 400** (nothing
unverified can ride the wire, and an `extraBody` typo fails loudly); `max_tokens` is clamped
not refused; `response_format: json_object` honoured; **`logprobs` refused**; no `seed` = the
engine picks; `n` defaults to `max(8, max-num-seqs)`; stop strings ≤ 64 × 4096 B.
- **Tools:** the qwen3.8 dialect from the chat template → `message.tool_calls` +
`finish_reason: tool_calls`. **`reasoning_effort`:** the froggeric template does not gate the
set (xhigh/high/medium/low all 200); the harness map is a safety net, not a requirement.
- **Vision is native** (the checkpoint's 27-layer Qwen-VL tower through the engine's Qwen-VL
processor): an `image_url` content part, no legacy top-level `images` field, ≤ 4,194,304 px
and 16,384 patches per pass. Probe with a ≥ 256 px solid-colour PNG and a ≥ 256 token budget
— at 64 tokens you get empty `content` and the colour only inside `reasoning_content`.
- **Capacity behaviour:** a prompt over `--max-model-len` is refused, never truncated; a request
longer than the KV pool's token count is accepted then refused as too large to fit; under the
3-slot contract a **fourth** full-height conversation is refused, not paged or squeezed.
- **Observability:** `/metrics` carries vLLM-compatible names plus `radiance:*`, scraped by the
Dankclaw Prometheus job at `192.168.0.235:7770` over plain LAN routing (the netavark
published-port quirk is same-host only). The boot ledger — the referee for every shape number
above — is in the container log (`podman logs frank-archon`);
`/home/dank/logs/frank_archon.log` is where the serve script redirects `podman run`.
## Files
- Shape and law owner: `scripts/frank-archon.env` (measured arithmetic in its header),
`scripts/serve-frank-archon.sh` (gates image / `.rad` size / template / GPU nodes, retires
stale containers, `podman wait`), `scripts/frank-archon.service`, `scripts/ttm-preflight.sh`.
- Payload doors: `scripts/download-frank-archon-iq4r.sh` (live), `scripts/download-frank-archon.sh`
+ `scripts/build-frank-archon-rad.sh` + `scripts/stack-qwen38-official-fp8.py` (the fp8-native
rebuild), `scripts/download-frank-archon-swift15.sh` (the BF16 source a w4dn8 re-convert
needs), `scripts/hfget.sh` / `scripts/hf-fetch.sh`; template
`scripts/qwen38-froggeric-v22.5.jinja`.
- Convert campaign (dormant, kept for a future quant): `scripts/convert-swift15-w4dn8.sh`,
`scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`, `scripts/run-swift15-bf16.sh`,
`scripts/kld-gate-swift15.sh`, `scripts/kld-then-calib.sh`, `scripts/calib-push.py`,
`scripts/make-calib-corpus.py`, `scripts/derive-protected-experts.py`,
`scripts/apply-protected-cast.sh`, `scripts/verify-swift15-ple.py`.
- Probes: `scripts/probe-frank-archon-kv.py` (reads slots/window/dtype from the deployed env so
it cannot drift; leg D = decode), `scripts/frank-archon-expert-activity.py`,
`scripts/autoresearch-frank-archon.sh` + `scripts/autoresearch/frank-archon-measure.py`.
- Deploy and tests: `scripts/deploy.sh` / `scripts/deploy-remote.sh` (`deploy_llm_archon`),
`tests/frank_archon_unit.bats` (the seams gating the KV-pool arithmetic, the VRAM pair and the
ttm gate), `tests/changed_tests.bats`, `tests/deployment_scoping.bats`.
- Retired-era leftovers in-tree: `scripts/patches/glm53-*.patch`,
`scripts/patches/gpu_worker-kv-cap.patch` (fork archaeology; none applies byte-exact today).
Related: [[frank-dragoon]] (the abliterated family), [[frank-zealot]],
[[Dual-R9700-Cooperation]], [[deployment]], [[swift15-campaign-worklog]],
[[dankserver-frontier-moe-feasibility]], [[x4-pair-experiment]], [[RDNA4 Container Serving]],
[[frank-medium-radiance-mxfp4]], [[LLM/index]].
serverAppConfigs-docs/LLM/frank-dragoon-benchmark.md
-128
View File
@@ -1,128 +0,0 @@
---
type: reference
description: "This page records the benchmarks that chose the runtime for `frank-dragoon` on Dankclaw."
tags:
- llm
- frank
timestamp: 2026-09-03
---
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
> See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
> [deployment.md](deployment.md) for the active configuration.
# frank-dragoon — Backend and Spec Benchmark
This page records the benchmarks that chose the runtime for
`frank-dragoon` on Dankclaw. Every number came from the live
llama.cpp build on that machine (upstream, b10133+).
## Summary
- **Backend**: **Vulkan0** (2026-08-15). At 262k ctx Vulkan decodes 48%
faster than ROCm (5.68 vs 3.83 t/s); ROCm only wins prefill 119 vs 106
t/s. Generation is the interactive cost, so Vulkan0.
- **Spec (2026-08-15)**: llama.cpp `--spec-default` preset (ngram-mod,
n_match 24 / n_min 48 / n_max 64, draftless). Replaced DSpark, which
crashed the ROCm backend under two concurrent 16k prompts (see
[[qwen3.8-spec-matrix]]).
- **Parallel**: 2 slots — passes the 2x32k parallel stream test.
- **Context**: 2 slots x 262144.
- **KV**: `q8_0` K / `q8_0` V on GPU.
- **Batch**: 8192 / ubatch 256.
> The A/B tables below are retained for history but predate the 2026-08-14
> Qwen3.8-27B `UD-Q8_K_XL` switch. The current production spec decision,
> the DSpark parallel crash, and the high-context ROCm-vs-Vulkan bench are
> recorded on [[qwen3.8-spec-matrix]].
## 2026-08-14: Spec A/B (ROCm, production config, default sampling, 1.1k-token prompt)
| Spec | Generation (t/s) | Parallel-safe |
| ---- | ---------------- | -------------- |
| DFlash (draft-dflash, n_max 8) | ~20.2 (from earlier 4k bench) | **NO** - premature EOS |
| MTP (draft-mtp, n_max 3) | **15.3 - 19.5** | YES |
| MTP + ngram-mod (n_max 3) | 12.0 - 13.6 | YES |
| MTP (draft-mtp, n_max 2) | 12.4 - 14.3 | YES |
| ngram-mod | 6.7 - 8.7 | YES |
Verdict: `draft-mtp` at `n_max 3` is the winner. It nearly matches DFlash's speed, beats ngram-mod by ~2x, and does not exhibit the DFlash parallel bug. Layering ngram under MTP (12-13.6 t/s) does not pay off, the same result as layering ngram under DFlash.
## Incident: DFlash Premature EOS Under Parallel Load (2026-08-14)
Reproduced three times by the user. When a new task lands on a slot with high prompt-cache reuse (LCP similarity ~0.99, `f_keep` ~0.98) while another slot is generating, the generating slot is released ~1 second later with `finish_reason "stop"` and a truncated response. The client records it as a normal completion (no abort). Server log shows a `slot release` with `truncated = 0` and no `cancel task` line.
Mechanism: DFlash's draft model uses one shared draft context (`ctx_dft`) across all parallel slots. When the new task's first DFlash draft pass runs after a large prompt-reuse operation, it corrupts the generating slot's draft state, and the target spuriously samples the EOG token. Known class of bug upstream (`server : do not default to multiple slots with speculative decoding`, PR #17017; DFlash multi-slot routing fixes in the beellama.cpp fork).
Fix: switch away from DFlash. ngram-mod (draftless) and MTP (separate draft machinery) both pass the strict parallel test. MTP was chosen for speed.
## Backend Comparison (llama-bench, raw target model, no draft)
Both backends ran the same target model at 64k context in `llama-bench`
(`-p 65536 -n 256`).
| Device | Prefill 64k (t/s) | Generation 64k (t/s) |
| -------- | ----------------- | -------------------- |
| ROCm0 | 220.28 | 6.99 |
| Vulkan0 | 72.34 | 7.00 |
ROCm prefills 3x faster. Generation is a tie in the raw benchmark.
## Full-Pipeline Comparison (4k context, real served chain)
These used `bench-pipeline-64k.sh` driving `llama-server` through the API at
4k context, reading the server's own `print_timing` lines.
| Spec | Backend | Prefill (t/s) | Generation (t/s) | Draft acceptance |
| --------------------------- | ------- | ------------- | ---------------- | ---------------- |
| `draft-dflash` | ROCm | 148.2 | **20.17** | 0.50 |
| `draft-dflash,ngram-mod` | ROCm | 149.9 | 16.93 | 0.42 |
| `ngram-mod` | ROCm | 154.0 | 8.68 | 0.57 |
| `ngram-mod` | Vulkan | 259.6 | 6.90 | 0.17 |
### Interpretation
- DFlash alone gives the fastest generation on ROCm (20.17 t/s).
- Layering ngram-mod lowered ROCm generation (16.93 t/s), so it is not
layered in production. llama.cpp *can* layer DFlash + ngram
(`--spec-type draft-dflash,ngram-mod`), but it was slower here.
- Vulkan only works without the DFlash draft. The DFlash `ctx_other`
setup aborts the Vulkan backend (`ggml-backend.cpp:898: pre-allocated
tensor ... cannot run the operation (NONE)`). Vulkan ngram generation is
also slowest (6.90 t/s) with poor acceptance (0.17).
**Verdict: ROCm + `draft-dflash`.**
## Memory Constraint at 602100 Context
At `3 x 200700` context the KV cache alone is ~50 GB and the model is
29.77 GB, leaving only ~16 GB of the ~96 GB ROCm allocation for compute
graphs. The compute graph scales with **ubatch**, so:
- batch 12288 / ubatch 4096 -> OOM (80.7 GB prefill buffer)
- batch 6144 / ubatch 2048 -> OOM (40.9 GB prefill buffer)
- batch 2048 / ubatch 2048 -> OOM (ubatch 2048 too large)
- batch 2048 / ubatch 512 -> **fits and loads**
The requested batch 12288 / ubatch 4096 is not feasible at this context
size on this hardware.
## Command to Reproduce
Raw backend bench (ROCm shown; swap `-dev` for the other):
```
llama-bench -m <main>.gguf -p 65536 -n 256 -c 65536 -dev ROCm0 -r 1
```
Full-pipeline bench:
```
./scripts/bench-pipeline-64k.sh ROCm0 draft-dflash
```
## See Also
- [[deployment]] — deploy and update process
- [[DFlash DSpark Speculative Decoding]] — DFlash / DSpark / layering details
- [[index]] — knowledge base index
serverAppConfigs-docs/LLM/frank-dragoon-ciru-orca.md
-16
View File
@@ -1,16 +0,0 @@
---
type: reference
title: "CIRU v4.4.1 Orca on Dankclaw (frank-dragoon)"
description: "The retired frank-dragoon runbook (2026-09-19 to 2026-09-21, rollback target) — CIRU v4.4.1 (QSA window hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca on the Dankclaw Strix Halo iGPU: the private host build, the runtime gate, the payload, the deployed config, MTP and PLE, the client contract, and the failure modes worth knowing."
tags:
- llm
- ciru
- dankclaw
- frank-dragoon
- strix-halo
- gfx1151
- qwen4exp
- rocm
- mtp
timestamp: 2026-09-19
---
serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md
-338
View File
@@ -1,338 +0,0 @@
# frank-dragoon — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 NATIVE abliterated (Dankclaw, Strix Halo iGPU)
**ACTIVE since 2026-10-17.** Rollback = revert the swap commit and redeploy
(`scripts/frank-dragoon.env`, `scripts/serve-frank-dragoon.sh`,
`scripts/frank-dragoon.service`); the Swift1.5 payload stays on disk at
`/home/dank/models/halogen-flashnext-v2/` (its fetch script
`scripts/download-frank-dragoon-flashnext.sh` stays the door back), and the
27B-era payload beside that at `/home/dank/models/halogen-qwen3.8-27b/`.
Era write-up: [[frank-dragoon-flash-server-v2-swift15]].
## Why this payload
Frank's ask: serve
`https://huggingface.co/Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated`
with the card's recommended settings, and **save memory**.
This is the **native 2026-10-11 v2 conversion** of orcarouter's pinned BF16
Qwen3.8-Flash-Next, abliterated — the base itself, converted straight into
the v2 `.hgn` format. It is **not** the writer transplant that carried this
filename before: the artifact measures **66,687,678,432 B** where the
Swift1.5 r2 measured 66,687,678,464 B — 32 bytes apart, same filename
family, different model. Byte counts and the Hub LFS sha256 are the only way
to tell them apart, which is why the fetch script and both gates check them.
What the swap buys, measured (table below): retrieval at depth, decode,
prefill, and **7.4 GiB of RAM handed back**. What it costs: the Swift1.5
finetune's compressed reasoning and its small perplexity edge.
## The companion law (the reason this swap needed a referee)
The release publishes **only the main file**. The n-gram lookup table
(REQUIRED — the engine reads it beside the checkpoint), the vision tower,
and the tokenizer are peonist's, and the card does **not** establish them as
interchangeable with this native conversion. No source-matched pair is
published anywhere, so the only available companions are the stock ones.
They are **hardlinked** out of the verified Swift1.5 payload directory
(`scripts/download-frank-dragoon-v2abl.sh`): a hardlink is a regular file, so
it satisfies the serve gate's no-symlink law, and it costs no disk and no
copy next to a 62 GB payload. The engine's own checkpoint tools are the
referee, and they answered: **NIAH 18/18** (vs 17/18 for the payload those
files were published with), needle retrieval exact at both depths, the red-PNG
vision probe exact, perplexity in the normal band. A wrong PLE table does not
whisper — see Verification.
## The payload — one new file, three inherited ones
`/home/dank/models/halogen-v2-abliterated/`, fetched by
`scripts/download-frank-dragoon-v2abl.sh` (`-f` + exact byte counts + LFS
sha256 + `--verify-only`; no token on the command line):
|File|Bytes|sha256|Source|
|---|---|---|---|
|`qwen38-flash-next-v2-abliterated.hgn`|66,687,678,432|`b5bd925f…`|Quat3rnion `@main`, fetched|
|`qwen38-flash-next-ngram.hgn`|51,200,246,144|`3450acada…`|peonist `@5cc17cea`, **hardlinked**, REQUIRED|
|`qwen38-flash-next-vision.hgn`|897,916,416|`d62e0ae5…`|peonist `@e053f488`, hardlinked, vision ON|
|`tokenizer/` (6 files)| — |`tokenizer.json` LFS|peonist `@e053f488`, hardlinked|
The engine stays **halogen-flash-server 0.16.2**, digest-pinned (the card
tested 0.15.0; 0.16.2 is newer in the same line and is what the v2 format was
qualified on in this fleet):
```
ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
```
## Shape — the card's settings, and the three deliberate deviations
`HALOGEN_CTX=200704` (per-request ceiling — the card's 262,144 lowered the
same evening, see below), `HALOGEN_KV_SLOTS=2`,
**`HALOGEN_KV_POOL_POSITIONS=401408`** (one pool behind both slots, raised to
2 × ctx on 2026-10-18 — see *Pool doubled for slot parity* below; the card
recommends it equal to ctx, and that equality is where the memory is),
`HALOGEN_SPEC_ADAPT=0`, `HALOGEN_MAX_TOK=8192`,
`HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` (coupled pair),
sampler `1.0 / 0.95 / 20`, `HALOGEN_WEIGHTS_LOCK=1`,
`HALOGEN_HOST_RESERVE_GIB=20`.
Deviations from the card, each on purpose:
- `HALOGEN_MODEL_ID=frank-dragoon`, not the card's `halogen-v2-abliterated` —
the fleet naming law (service = unit = served id).
- `HALOGEN_SPEC_ADAPT=0` — the card is silent; this is a measured 0.16.2 law
(the adaptive default disables the drafter per request and halves decode).
- the sampler values are peonist's engine recipe; Quat3rnion publishes none.
**The pool is one budget, not a partition.** Boot ledger form: `listening on
127.0.0.1:8730, 2 slots over ONE <pool>-position KV pool, each request up to
<HALOGEN_CTX>`. Both numbers were lowered to **200,704** the same evening
(the card ships 262,144; Frank took the RAM — nothing else in the fleet asks
for the bigger window), so `models.yml` `contextWindow` moved to **155,648**
= ceiling − 45k with it. With pool == ctx, one maximum-length request fills
the pool: **two long conversations can no longer share the endpoint** (two of
≤100k can). That is acceptable here because frank-dragoon carries the `slow`
(medium-thinking) role and the `security-reviewer` subagent, not the
interactive workhorse (that is frank-archon).
Frank lifted exactly that concurrency limit on 2026-10-18 and the lift is
**deployed** (section below): the pool is again `slots × ctx`, so two
maximum-length conversations fit the endpoint at once.
**Memory ledger** (121.2 GiB unified), from the boot ledgers. The 22:20 boot
printed `pinned 62.11 GiB in 1 range(s) in 1.6 s (41.0 GB/s)` trunk,
`0.84 GiB` vision tower, `reserving the KV pool, 262144 positions (about
7.2 GiB)`, and the engine's own sum: `memory: 62.9 GiB of weights locked in
RAM (0.8 of them the vision tower's), 7.2 GiB of KV pool, …, 75.8 GiB in
all`. The cut boot printed the same shape at the new size: `reserving the KV
pool, 200704 positions (about 5.5 GiB)` and `… 5.5 GiB of KV pool, 5.7 GiB of
working memory, 74.1 GiB in all`.
The n-gram table rides as page cache — `mapped 51.2 GB,
NOT pinned (Pin::None)` — which is exactly why every pool cut is a *double*
saving. FLAGS.md law: **RAM the pool takes is RAM that table loses**; the ~9
GiB the two cuts together handed back is page cache for the 47.68 GiB table
that long prompts read.
`HALOGEN_FLASH_PIN_TRUNK=0` is **rejected by the v2 loader** — the trunk
stays pinned. Disk-backing the weights is not a lever on this format.
## Pool doubled for slot parity — committed AND deployed 2026-10-18
`HALOGEN_KV_POOL_POSITIONS` **200,704 → 401,408** (commit `2b12716`), Frank's
call: frank-dragoon's total in-flight context was **half** frank-zealot's —
the llama.cpp endpoint runs one 401,408-position pool split into two 200,704
conversations, while dragoon ran one 200,704-position pool behind its two
slots, so one maximum-length request filled it and any second long
conversation had to stay under ~100k (commit message;
`scripts/frank-dragoon.env:26-27`, `:183-186`).
- **The per-request ceiling does NOT move.** `HALOGEN_CTX` stays 200,704, the
native wall stays 262,144 (`HALOGEN_ROPE_YARN` law), and `models.yml`
`contextWindow` stays **155,648** — the pool is the shared budget behind the
slots, not the window. The serve seam enforces `HALOGEN_CTX ≤
HALOGEN_KV_POOL_POSITIONS ≤ HALOGEN_KV_SLOTS × HALOGEN_CTX`
(`scripts/serve-frank-dragoon.sh:85-95`); 401,408 is `slots × ctx`, the top
of the legal band, and exactly frank-zealot's pool number — parity, not
approximation.
- Measured reservation **11.0 GiB** — the boot ledger prints `reserving the KV
pool, 401408 positions (about 11.0 GiB)`. The preflight KV-budget line bills
**11.3 GiB** (`~28 KiB/position incl. block scratch + ~115 MiB/slot`) plus
1.3 GiB of prompt cache. The measured points beside it: 5.5 GiB at 200,704,
7.2 GiB at 262,144, and 14.6 GiB at the 2026-10-06 shape's 524,288
(`scripts/frank-dragoon.env:49-52`, `:74-76`).
- That RAM comes out of the n-gram table's page cache — FLAGS.md law: RAM the
pool takes is RAM that table loses.
**Deployed 2026-10-18** (`./scripts/deploy.sh dankclaw --service
frank-dragoon`, Playwright gate 33/33). The boot ledger confirms the shape:
`reserving the KV pool, 401408 positions (about 11.0 GiB)`,
`2 slots over ONE 401408-position KV pool, each request up to 200704 (it
reserves prompt + max_tokens)`, and the sum
`62.9 GiB of weights locked in RAM (0.8 of them the vision tower's), 11.0 GiB
of KV pool, 5.7 GiB of working memory, 79.6 GiB in all`. Prompt cache ON (12
entries × 111 MiB at ctx 200704 = the 1.3 GiB in the KV-budget line), n-gram
47.68 GiB still disk-backed (`Pin::None`), no clamp line. Host `MemAvailable`
fell 26 → 22 GiB across the raise, which is the FLAGS.md collision being paid
in page cache. **Warm reuse survived it — measured after the deploy.** One
**101,987**-token wikitext prompt through `/v1/chat/completions` took **81.5 s**
cold (1,251 t/s of prefill) and **0.3 s** on each of two repeats:
`HALOGEN_CACHE_INPLACE=1` keeps the KV rows in the pool, so a warm turn prefills
nothing and the doubling evicted nothing the cache needed. (The pre-raise
136.7× figure was a `ttft` reading; this engine sends **no `Server-Timing`**
header and has **no `/completion` path** — `/ping`, `/props`, `/tokenize` are
404 too — so wall-clock around the POST plus `usage.prompt_tokens` is the only
instrument available here.)
## Measured A/B — same battery, both boots (2026-10-17)
Behavior probes ran against the live service (tagged `swift15` / `v2abl`);
the engine tools ran with the service **stopped**, per the fleet benchmarking
law, under one identical knob set (`scripts/census-frank-dragoon.sh`).
|Probe|Swift1.5 r2 (out)|native v2 (in)|
|---|---|---|
|`verify` on the file|—|OK|
|NIAH 18 cases (4k/32k × 3 depths × 3 positions)|17/18 (94.4 %)|**18/18 (100 %)**|
|wikitext-2 PPL (297,053 ids, 2 × 262,144 seqs)|**3.8772**|3.9679 (+2.3 %)|
|KL of the new model against the old model's logits|—|mean 0.155 / median 0.057, top-1 85.1 %|
|refusal census (5 hostile prompts)|0/5 refusing|0/5 refusing|
|thinking tokens, mean of 6 questions (temp 1.0)|**784**|1,476 (×1.9)|
|tool call, 1-turn / 2-turn|`{"city": "Seattle"}` / answer|`{"city": "Seattle"}` / answer|
|vision (512 px solid red, `image_url` part)|Red|Red|
|needle @21,568 tokens, planted 0.85 / 0.10|hit / hit|hit / hit|
|decode, greedy, 512 tokens|46.8 t/s (spread 25.6–46.8)|50.0 t/s (steady 50.0/50.2)|
|prefill, cold, 59,024 tokens|1,424.8 t/s|**1,604.3 t/s (+12.6 %)**|
|prompt-cache ratio (same prompt, second pass)|136.7×|131.6×|
|KV pool|14.6 GiB (524,288 positions)|**7.2 GiB (262,144)**|
Read the KL honestly: it is a **model-vs-model** divergence, not the
quant-vs-source comparison the tool was built for. Two different models on
the same text *should* differ; median 0.057 with 85 % top-1 agreement is the
signature of two sound models that disagree on content, and the worst rows
are date/name completions. A n-gram table mismatched to the checkpoint
corrupts the forward pass and shows up as elevated KL **everywhere** plus
degraded retrieval — which NIAH (18/18) and the needle probes rule out.
The decode line needs care: the baseline's own spread was 25.6–46.8 t/s
because its samples were taken while the 62 GB fetch was still streaming into
page cache, and the candidate's **first** sample after the battery's long
prompts read 32.8 before settling at 50.0/50.2. Honest claim: **no decode
regression, plausibly faster**; never quote either single number.
## Verification — how to reproduce
```
# 1. payload: bytes + LFS sha256 + the hardlinked trio
bash scripts/download-frank-dragoon-v2abl.sh --verify-only
# 2. engine tools on a STOPPED service (fleet law: no GPU experiment beside
# a live endpoint; stop the unit first — mask fails on a relinked unit)
bash scripts/census-frank-dragoon.sh verify # no GPU, safe while serving
sudo systemctl stop frank-dragoon
bash scripts/census-frank-dragoon.sh ref # reference dump, old payload
bash scripts/census-frank-dragoon.sh cand # PPL + KL/top-1 vs that dump
bash scripts/census-frank-dragoon.sh niah-cur # baseline retrieval
bash scripts/census-frank-dragoon.sh niah-new # candidate retrieval
# 3. boot the candidate through a transient drop-in, then the battery
bash scripts/trial-frank-dragoon.sh candidate
python3 scripts/battery-frank-dragoon.py v2abl all
bash scripts/trial-frank-dragoon.sh restore # back onto the repo env
```
The battery needs two staged assets in `/tmp/halogen-eval/` (a 512 px solid
red PNG as base64, `red512.b64`, and the wikitext-2 raw test split,
`wikitext-2-raw-test.txt`). Laws it encodes:
- **the engine bills wikitext at ~4.33 characters/token** — the harness
chars/4 estimate under-bills by ~8 %, and a prompt over `HALOGEN_CTX` is
refused with a 400 that names the exact overage (never truncated). Size
haystacks in characters against a **token** budget.
- the wikitext-2 raw test split as staged joins rows with a **single**
newline (2,891 lines, zero blank lines), so a blank-line split returns one
1.29 MB "paragraph" and every budget silently becomes "everything".
- the first request after a boot is a warmup artifact; the battery discards
it (`warmup` row) and never measures it.
- `ppl`/`niah` **load the model** → stopped service. `verify` reads the file
back with no GPU and can run while serving.
## Contract (unchanged from the Swift1.5 era)
`reasoning_content` carries the thinking and counts against `max_tokens`
(≥512-token budget on any vision probe or the endpoint looks blind while it
is mid-thought); vision via OpenAI `image_url` content parts only, ≥256 px
probes; `response_format` unsupported; the chat template rejects
`reasoning_effort` outside `xhigh|medium|low` (the `models.yml` fold stays
mandatory); unknown sampling params are a 400; the prompt cache is on by
default, so a warm 100k follow-up costs seconds (measured after the pool raise:
**81.5 s cold vs 0.3 s warm** at 101,987 prompt tokens). The HTTP surface is
thin — `/health`, `/metrics`, `/v1/models` and POST `/v1/chat/completions`
answer, while `/ping`, `/props`, `/completion` and `/tokenize` are 404 and no
response carries `Server-Timing`: llama.cpp's timing instruments do not exist
on this engine, so measure wall-clock plus `usage.prompt_tokens`. See
[[frank-dragoon-flash-server-v2-swift15]] for the engine-flag detail that
carries verbatim, and [[deployment]] for the deploy/health gate.
## Production boot — the deploy ledger (2026-10-17 22:20, `fcd7630`)
`./scripts/deploy.sh dankclaw --service frank-dragoon`: healthy after ~12 s,
deploy marker `fcd7630`, e2e gate 33/33. The trial drop-in went to
`~/.AGENTTRASH/20261017-v2abl-trial/` FIRST, so this boot is governed by the
repo env — `systemctl show -p DropInPaths` is empty.
Ledger lines that settle the shape (engine stdout, not the journal):
|Line|Figure|
|---|---|
|preflight|`MemAvailable now 105.4 GiB` against the engine's own estimate of 62.1 weights + 6.0 working + 7.4 KV + 1.3 prompt cache|
|weights|`62.9 GiB of weights locked in RAM (0.8 of them the vision tower's)`|
|KV pool|`reserving the KV pool, 262144 positions (about 7.2 GiB)`; `the KV is ONE pool of 262144 positions (6656.0 MiB) carved into per-request regions`; slots `1 -> 2 POOLED, 111.0 MiB each of O(1) state`|
|working memory|5.7 GiB at `HALOGEN_MAX_TOK 8192`|
|total|`75.8 GiB in all` — ~25 GiB of the 121.2 GiB box left for co-tenants|
|n-gram|`47.68 GiB from /models/qwen38-flash-next-ngram.hgn`, `checkpoint: mapped 51.2 GB, NOT pinned (Pin::None)` — disk-backed exactly as designed|
|prompt cache|ON, resume-anywhere, 12 entries, chunk 32768, `HALOGEN_CACHE_INPLACE=1` (the KV rows are never copied)|
|spec decode|`MTP depth-2 selectable (drafter 1, speculates while alone)`|
|clamps|none — the requested pool is the delivered pool|
Read the two KV figures together: the engine's preflight FORMULA bills 7.4 GiB
(~28 KiB/position **including** block scratch and ~115 MiB/slot of O(1) state)
while the RESERVATION is 7.2 GiB / 6,656 MiB. Quote the reservation for the
RAM ledger and the formula when comparing pool sizes.
Live probes on the deployed endpoint: `27*59` → `1593` with a warm thinking
channel, tool call `{"city": "Seattle"}`, 512 px solid-red PNG → **Red** (320
prompt tokens — the image reached the model), greedy 512-token decode 46.9 t/s
on the FIRST request after boot, i.e. the warmup artifact; the trial's settled
figure is 50.0. OpenWebUI's `frank-dragoon` row description re-keyed to the
native payload (backup `webui.db.bak-20261017b`; no container restart forced —
forcing it logs every family session out for one cosmetic string).
## Ceiling lowered to 200,704 — the second production boot (`7c4dbeb`)
The same evening, after the ledger above was read: `HALOGEN_CTX` and
`HALOGEN_KV_POOL_POSITIONS` 262144 → **200704** (49 × 4096, the per-slot rung
the llama.cpp endpoint frank-zealot already runs). The checkpoint's native
window is 262,144 and the engine serves anything below it, so this is a RAM
decision, not a capability limit. Nothing in the fleet asks for the bigger
window; the harness moved WITH the ceiling (400-zone law), `models.yml`
`contextWindow: 155648`.
|Line|Figure|
|---|---|
|marker|`revision=7c4dbeb…`, `status=deployed`; unit `active`, `/health` 200|
|KV pool|`reserving the KV pool, 200704 positions (about 5.5 GiB)`|
|shape|`2 slots over ONE 200704-position KV pool, each request up to 200704 (it reserves prompt + max_tokens), max_tok 8192 … MTP depth-2 selectable (drafter 1…)`|
|working|5.7 GiB (unchanged — it scales with `HALOGEN_MAX_TOK`, not the window)|
|total|`62.9 GiB of weights locked … 5.5 GiB of KV pool, 5.7 GiB of working memory, 74.1 GiB in all`|
|prompt cache|ON, resume-anywhere, 12 entries, chunk 32768, **111 MiB each at ctx 200704**|
|clamps|none — requested pool == delivered pool|
The refusal contract at the new ceiling, asked for explicitly (the 1.29 MB
wikitext corpus through `/v1/chat/completions`):
```
HTTP 400 {"error":{"message":"the prompt does not fit: it is 297109 tokens and
the context is 200704, 96405 tokens over. Shorten the prompt, or raise the
context (HALOGEN_CTX, and HALOGEN_ROPE_YARN for more than the native 262144).",
"type":"invalid_request_error"}}
```
Refused in 0.86 s — the preflight rejects before any prefill, so probing the
ceiling costs the endpoint nothing. OpenWebUI's `frank-dragoon` row description
re-keyed to "ONE 200,704-position KV pool at ctx 200,704" (still no container
restart; `webui.db.bak-20261017b` holds the pre-cutover row).
**What the cut cost:** with pool == ctx one maximum-length request fills the
pool, so two 200k conversations never coexist (two of ≤100 k do), and the
harness window is now 155,648 — the 180 k compaction target of the 262,144 era
is unreachable because the window itself is smaller. (That pool == ctx
concurrency limit IS superseded: the doubling deployed the next day — see
*Pool doubled for slot parity* above. The window math stands, though: the pool
never moves the per-request ceiling.)
**What it bought:** 1.7
GiB back as page cache for the 47.68 GiB n-gram table (9.1 GiB against the
2026-10-06 shape's 524,288 pool), and a smaller prompt-cache entry.
**Third production boot — the `401408` pool, deployed 2026-10-18.** Same ctx,
same working set, same 111 MiB prompt-cache entry; the pool line reads
`reserving the KV pool, 401408 positions (about 11.0 GiB)` and the total
`79.6 GiB in all`. Ledger and the measured cost in
*Pool doubled for slot parity* above; host `MemAvailable` after that boot:
22 GiB.
serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-swift15.md
-197
View File
@@ -1,197 +0,0 @@
# frank-dragoon — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, Strix Halo iGPU)
**RETIRED 2026-10-06 → 2026-10-17**, superseded by the native v2 conversion:
[[frank-dragoon-flash-server-v2-native]]. Rollback target — the payload
stays on disk at `/home/dank/models/halogen-flashnext-v2/`,
`scripts/download-frank-dragoon-flashnext.sh` re-verifies it, and restoring
the era is a revert of the 2026-10-17 swap commit. The 27B payload this era
replaced stays at `/home/dank/models/halogen-qwen3.8-27b/` with
`scripts/download-frank-dragoon-halogen.sh`. Everything below is the era as
it shipped; the engine-flag detail still governs the current page.
## Why this shape exists
Frank's ask: replace the frank-dragoon model with
`https://huggingface.co/Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated`.
That payload is an **abliterated Swift1.5 finetune of peonist's
Qwen3.8-Flash-Next v2 checkpoint in the v2 `.hgn` format** — a file that
loads ONLY in the **halogen-flash-server** engine line, NOT in the halogen
0.1.4 container that served the 27B dense checkpoint. The model move is
therefore a **full engine migration**, and the accepted-product facts flip
twice in the fleet's favour:
- the base is **abliterated** (the 27B halogen endpoint was a finetune of
the censored `Qwen/Qwen3.8-27B`; refusals were an accepted regression there),
- **vision is back** — the v2 vision tower loads beside the checkpoint and
the engine runs it (the 27B halogen engine silently ignored its encoder),
- **MTP speculative decoding returns** with the head inside the checkpoint
itself (the 27B shipped a DFlash2 drafter instead; comparable purpose,
different file plumbing),
- the card claims **53–57 t/s greedy decode, ~1,640 t/s prefill at 34k, ~72 %
fewer thinking tokens** than the censored 27B (905 mean), and **262,144
native context**. Publisher-card numbers — verify against live traffic
before quoting further (see Verification).
The engine is **closed-source**, ships as `ghcr.io/peonist-ai/
halogen-flash-server:0.16.2` (the version the model card's quickstart pins),
digest-pinned:
```
ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
```
(3.6 GB image; RepoDigest captured at pull 2026-10-06. Card-tested fallback
if a future pin misbehaves: 0.15.1.)
## The payload — a triangle of files, all sha256-verified at fetch
| File | Bytes | Source repo @ revision | sha256 (LFS) |
|---|---|---|---|
| `qwen38-flash-next-v2-swift15-abliterated.hgn` | 66,687,678,464 | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated @ main (`69a85db`) | `9c880b8ba48deabfa06172eff1e7572e4fe465e09e97faec6e50c3f6fdcac037` |
| `qwen38-flash-next-ngram.hgn` (REQUIRED) | 51,200,246,144 | peonist-ai/halogen-qwen3.8-flash-next @ `5cc17cea` | `3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe` |
| `qwen38-flash-next-vision.hgn` (enabled) | 897,916,416 | peonist-ai/halogen-qwen3.8-flash-next @ `e053f488` | `d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10` |
| `tokenizer/` (6 files, flat dir) | 22.4 MB total | peonist @ `e053f488` | `tokenizer.json` is LFS `0997f410…` (byte-identical to the 27B-era copy); the rest are plain git blobs, size-checked |
Payload notes worth keeping:
- The deployed checkpoint is a **compacted r2** of the card's append-only
original (89,775,868,032 B → 66,687,678,464 B): the abliteration workflow
appends and the repo's `tooling/` compact drops dead padding; live tensors
are byte-identical, the LFS sha256 is the contract, and the 24 GiB saving
is why it fits.
- **`HALOGEN_MTP_HEAD` stays UNSET**: the MTP head is INSIDE the v2
checkpoint (the separate `qwen38-flash-next-mtp.hgn` sidecar is for a
different assembly and is deliberately not fetched).
- The **n-gram lookup table is REQUIRED**. The engine reads it beside the
checkpoint (documented fallback); the card's quickstart also names it
explicitly via `HALOGEN_NGRAM_TABLE`, and the serve script passes both —
the env var is belt-and-braces over the documented beside-the-checkpoint
default, and both mechanisms resolve to the same file.
- The tokenizer is fetched fresh from peonist @ `e053f488` into
`/home/dank/models/halogen-flashnext-v2/tokenizer/` — same flat-directory,
no-symlink law as every other payload (an HF cache snapshot's symlinks
dangle inside the container mount; both the serve gate and the deployer
refuse them).
Fetch: `scripts/download-frank-dragoon-flashnext.sh` (`--verify-only` to
re-check; `-f` + exact byte counts + LFS sha256 + resolve-the-CDN-then-`-C -`
resume law throughout; no token — the repos are public).
## Shape — the knobs that are not free
`scripts/frank-dragoon.env` carries the full rationale inline; the table:
| Env | Value | Why |
|---|---|---|
| `HALOGEN_CTX` | 262144 | per-slot HARD ceiling (the checkpoint's native window; a longer prompt is a 400 naming the limit, never truncation). **RAISED from 200192 on 2026-10-07 — the 400-zone law:** the harness clamps `max_tokens = min(45056, contextWindow − est − 4096)`, its chars/4 est UNDER-counts the rendered prompt by ~3 % (measured twice: est 152,938 → server 157,388, request over room by 354; est 158,906 → server 165,301, over by 2,300), and compaction only fires at 85 % of the harness window — so with window == engine ceiling, every request in the est ~146k–184k zone was a guaranteed 400. Harness window 217,088 = ceiling − 45k absorbs the error: worst-case request ≈ 219k ≤ 262,144 (≈10 % under-count headroom) |
| `HALOGEN_KV_SLOTS` | 2 | the 27B engine's "slots>1 disables the drafter" law does NOT carry over — on this engine batching and speculation coexist (the card's quickstart ships 2 slots WITH the head). 2 slots = concurrent capacity |
| `HALOGEN_KV_POOL_POSITIONS` | 524288 | explicit form of the default (2 × ctx), so the seam test pins the product; ~28 KiB/position ⇒ 14.0 GiB KV |
| `HALOGEN_HOST_RESERVE_GIB` | 20 | keeps the host honest (the w4b-era lesson) |
| `HALOGEN_SPEC_ADAPT` | **0** | MANDATORY (card): the default adaptive policy turns the MTP head off for whole requests after low acceptance and **halves decode**. The head stays on; temp-1.0 traffic pays some acceptance for it — accepted trade |
| `HALOGEN_MAX_TOK` / `HALOGEN_MAX_TOKENS_CAP` / `HALOGEN_QUEUE_TIMEOUT` | 8192 / 65536 / 7200 | 8192 is only the per-request DEFAULT (the harness sends at most its `maxTokens` 45,056); the cap/timeout pair is COUPLED per FLAGS.md (65536 at ~10 t/s serial needs > 6550 s) |
| `HALOGEN_TEMPERATURE` / `TOP_P` / `TOP_K` | 1.0 / 0.95 / 20 | the card's thinking recipe as SERVER-SIDE defaults — the engine HAS a sampler now, so an absent temperature is no longer greedy; models.yml extraBody matches it (consistent, not load-bearing) |
| `HALOGEN_WEIGHTS_LOCK` | 1 | card's belt-and-braces; the engine pins weights by default on v2 (`HALOGEN_FLASH_PIN_TRUNK=0` is a w4b-only knob and is REFUSED by v2) |
| `HALOGEN_MODEL_ID` / publish | **frank-dragoon** / 7770:8731 | served id renamed 2026-10-09 (service/unit/env keep `frank-dragoon`); engine port 8730 never published |
| MTP depth | default 2 | FLAGS.md: 2 is best all-round, 3 for code — Frank's traffic is agent/prose |
**Memory ledger** (121.2 GiB unified, the referee is the boot ledger):
weights ~62 GiB resident + KV 14.0 + arena ~8 (`3.8 + 8.4 × 8192/16384`)
+ vision ~2 ≈ **86 GiB engine-side**, + 20 host reserve = ~106 with
~15 GiB margin over the box's baseline co-tenancy. The 51.2 GB n-gram
table rides as mmap'd page cache beside the checkpoint — the card's own
quickstart arithmetic does not bill it as resident, but WATCH the ledger
at boot and re-size if the engine prints a clamp. Prompt cache is ON by
default (`PROMPT_CACHE=2`): a follow-up turn at 100k ctx costs ~2 s
instead of ~88 s cold — the big-TTFT win the 27B engine needed a manual
`HALOGEN_CACHE_MB` pin for is built in here.
## Engine contract facts (FLAGS.md of the flash-server repo)
- OpenAI chat **and** `/v1/responses` **and** `/v1/messages` **and**
`/v1/completions`; tool calls verified on the card; `response_format`
still unsupported (no constrained decoding).
- Thinking → **`reasoning_content`** (same field as every engine this
endpoint has run); the v2 chat template rejects `reasoning_effort`
outside `xhigh|medium|low` — the models.yml fold (high→xhigh) stays
mandatory.
- Vision: OpenAI `image_url` content parts only (the fleet's standard
contract; the legacy top-level `images` field is the frank-zealot trap).
≥256 px probe images or the model answers "video".
- `HALOGEN_MAX_TOKENS_CAP` over-cap is a 400 naming the limit — never a
truncation. `max_tokens` INCLUDES thinking.
- All flags read once at startup; every flag is BITWISE except
`HALOGEN_CK_OVERLAY`, `HALOGEN_MATMUL_TUNING_FILE`,
`HALOGEN_INDEXER_BUDGET`, `HALOGEN_COMPOSABLE_CONTEXT`. Unset
`HALOGEN_CHECKPOINT` defaults to `/models/qwen38-flash-next-v2.hgn`
with the lookup table beside it.
- Prompt-lookup drafting (3,3 window) rides beside the MTP head by
default and is byte-identical to greedy verification where it fires.
- Watchdog: `HALOGEN_ENGINE_WATCHDOG_S=180` with defer-logic for memory
stalls — on a shared host the defaults are the safe choice; the
restart policy stays real through the root unit's `Restart=always`.
Kernel: `CONFIG_HSA_AMD_SVM=y` (CachyOS ships it); `amdgpu.noretry=-1`
(auto) — for this IP auto resolves to retry-on, the same effective state
the 27B engine ran on. Only revisit if the ledger shows SVM complaints.
## Mechanics
- **Root systemd unit** (`scripts/frank-dragoon.service`, `User=dank`,
`LimitMEMLOCK=infinity`, `Restart=always`, `ExecStop=podman stop -t 60
frank-dragoon-halogen`), driven by `scripts/serve-frank-dragoon.sh` — the
fleet's `podman run`-direct pattern. The script runs the container
FOREGROUND so the boot ledger (weights resident, pool fit, MTP head,
prompt cache, any clamp) lands in `/home/dank/logs/frank_dragoon.log`.
- Serve gates mirror the deployer EXACTLY (2026-10-01 crash-loop law):
image digest in podman storage, byte-exact checkpoint + n-gram table +
vision tower, flat tokenizer with all six files as regular files,
GPU nodes present.
- **Host-RAM pin-gate preflight** (added 2026-10-06): the engine
refuses to pin when `MemAvailable` is below the checkpoint (62.11 GiB)
plus a **16 GiB floor** — pinned pages cannot be swapped, and under the
floor the n-gram lookup pages in from disk on every long prompt. A
RAM-short host crash-loops the unit (each boot dies in ~5 s at the pin
gate) until the RAM clears; the 2026-10-06 cutover hit this (8 boots,
MemAvailable 74.77 GiB vs the 78.11 GiB needed, self-resolved in ~2 min
when the dying 27B engine freed RAM). The serve script now checks
`MemAvailable` BEFORE `podman run` and WAITS up to `FRANK_DRAGOON_RAM_WAIT`
(default 600 s) for the RAM to clear, so a transient dip boots once
instead of crash-looping; if it never clears it refuses loudly. The 16
GiB floor is the engine's (not `HALOGEN_HOST_RESERVE_GIB=20` — a
separate, higher, overall-budget knob).
- Engine stdout goes to the log, NOT the journal. The FIRST request after
a (re)start can sit silent ~45–60 s on this engine class (health
`engine_unresponsive`, 503s) and self-recovers — do not restart over it.
## Verification checklist (run after any redeploy)
1. Boot ledger greps in `/home/dank/logs/frank_dragoon.log`: resident
weights (~62 GiB), KV pool line, MTP/drafter line, prompt-cache line,
**no clamp named**.
2. `curl http://192.168.0.143:7770/health` → 200.
3. Text completion probe: reasoning lands in `reasoning_content`,
`timings` honest (generate ≥128 tokens; at 8 the inter_token number is
first-token noise).
4. Vision probe: a 512×512 solid-red PNG through an `image_url` part →
"Red" (≥256 px, one word expected).
5. Spec decode: `draft_n`/`draft_n_accepted` present in timings on
real-shaped traffic (SPEC_ADAPT=0 means the head is always engaged).
## Client contract (harness)
`~/.omp/agent/models.yml` — provider `llamacpp (dankclaw)` @
`http://192.168.0.143:7770/v1`: `contextWindow: 217088` — deliberately
BELOW the 262,144 engine ceiling: the 45k gap absorbs the harness's
~3 % chars/4 est under-count (see the `HALOGEN_CTX` row above) —
`maxTokens: 45056`, `input: [text, image]`,
`reasoningContentField: "reasoning_content"`, the `high→xhigh` effort
fold, `extraBody` temperature 1.0 / top_p 0.95. Compaction still rides
frank-zealot (the `compactionModel` steering is unchanged; the input
≈ 170k fits its 200,704 slot).
## Rollback
Revert the swap commit and redeploy — the 27B trio is intact on disk
(payload at `~/models/halogen-qwen3.8-27b/`, fetch script, and the halogen
0.1.4 image digest still in podman storage). The older ROCmFPX rollback
door (`build-rocmfpx.sh` pin) is untouched.
serverAppConfigs-docs/LLM/frank-dragoon-gufo-qwen38-27b.md
-350
View File
@@ -1,350 +0,0 @@
---
type: reference
title: "gufo, Qwen3.8-27B UD-Q4_K_XL (DFlash2) on Dankclaw (frank-dragoon)"
description: "The RETIRED frank-dragoon runbook (rollback target) — gufo (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source on Dankclaw and serving unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL (vanilla) with the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU, ONE session at the full native 262,144 window with unquantized f16 KV: the from-source build and its two host workarounds, the GGUF header evidence, why the fleet's Qwen-Sharp chat template CANNOT be used on this runtime, the client-contract delta, the measured prefill/decode trade, and the failure modes worth knowing."
tags:
- llm
- gufo
- swift-1.5
- dankclaw
- frank-dragoon
- strix-halo
- gfx1151
- q8
- dflash2
- rocm
timestamp: 2026-09-28
---
# frank-dragoon — gufo, Qwen3.8-27B (UD-Q4_K_XL, DFlash2)
> **RETIRED 2026-10-02 — the rollback target.** Superseded by
> [[frank-dragoon-rocmfpx-q8agent-mtp]] (Q8_0_ROCMFPX_AGENT at 8.39 bpw with
> MTP4, 17.0 t/s measured). This era was itself a return after one day of
> [[frank-dragoon-vllm-mimo-v26-bf16]] (battery decode 6.19-9.01 t/s; every
> ≥6-bit vLLM quant path dead on gfx1151). The shape
> moved on the way back: payload is now **unsloth/Qwen3.8-27B-GGUF
> UD-Q4_K_XL** (16.35 GiB, vanilla — the abliterated Swift Q8 was
> reclaimed), and the session grid is **one session at the full native
> 262,144 window** (`SESSIONS=1 CONTEXT_SIZE=262144`, f16 KV). The
> Swift-1.5 Q8 history below is retained for the engine/template law,
> which still governs.
> **Originally ACTIVE 2026-09-28.** Replaces
> [[frank-dragoon-unsloth-qwen38-27b-q8xl]] (stock llama.cpp, the unsloth
> b11030 prebuilt), which had replaced
> [[frank-dragoon-kairic-edge-qwen38-27b]]. Same host, same port (7770), same
> alias (`frank-dragoon`).
>
> **The payload also changed on 2026-09-28**, later the same day: the target
> went from `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` to
> `ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`'s
> `UD-Q8_K_XL` — an abliterated (uncensored) fine-tune of the same base. See
> [Payload](#payload) and, for what that costs, [the template
> law](#the-template-law-qwen-sharp-is-gone).
## Identity
- **Endpoint:** `frank-dragoon` at `192.168.0.143:7770` (Dankclaw).
- **Runtime:** [gufo](https://github.com/gufo-org/gufo) — "the Strix Halo
inference engine", a vertical engine built for exactly this hardware
(Ryzen AI MAX+ 395 / Radeon 8060S, `gfx1151`, unified memory). Pinned at
**v0.1.1** (commit `b0f8673`), built from source on the host to
`/home/dank/GitPackages/gufo-v0.1.1/`.
- **Service shape:** root systemd unit `scripts/frank-dragoon.service`
(`User=dank`, `LimitMEMLOCK=infinity`), launched by
`scripts/serve-frank-dragoon.sh` with tunables from
`scripts/frank-dragoon.env`.
gufo ships **no prebuilt Linux binaries** (the v0.1.0/v0.1.1 release assets
are empty) and no Nix on this host, so the runtime is a source build. It is
also young: v0.1.0 landed 2026-09-28, the same day as this cutover.
## Why this engine
gufo's own measurement of the two engines on the same Qwen3.8-27B targets
(Strix Halo, `628ed18e` vs llama.cpp `b11069`, greedy, DFlash2):
| Q8_K_XL, DFlash2 | gufo | llama.cpp | delta |
|---|---:|---:|---:|
| prefill, depth 0 | 457.05 t/s | 333.85 t/s | **+36.9 %** |
| prefill, depth 131,072 | 237.24 t/s | 131.80 t/s | **+80.0 %** |
| decode, depth 0 (mixed) | 16.08 t/s | 15.00 t/s | +7.2 % |
| decode, depth 32,768 (mixed) | 14.41 t/s | 14.44 t/s | −0.2 % |
| decode, depth 65,536 (mixed) | 10.87 t/s | 12.62 t/s | **−13.9 %** |
| decode, depth 131,072 (mixed) | 8.87 t/s | 11.62 t/s | **−23.7 %** |
| cold load to HTTP-ready | 8.17 s | 36.47 s | **4.5× faster** |
| peak memory @ 262,144 ctx | 50.58 GiB | 48.60 GiB | +4 % |
Read that honestly: **prefill, load time and short-context decode improve;
deep-context decode on the 8-bit target gets worse.** The crossover on Q8 is
around 32k of cached depth. frank-dragoon runs one 262,144-token conversation,
so a long agent session spends its life on the losing side of that line
while a short turn wins. The Q4_K_XL target does not show the regression
(gufo is +11.7 % to +17.0 % at every depth) — that is the honest reason to
revisit the quantization separately from the runtime.
## Build
`scripts/build-gufo.sh` reproduces the runtime end to end. It clones the
pinned tag, configures the `release` preset (HIP, `--offload-arch=gfx1151`),
builds, installs, and finishes with `gufo diagnose`. Two host facts it
encodes, both found the hard way on 2026-09-28:
- **Ubuntu's `librocwmma-dev` must not be installed.** It ships a truncated
rocWMMA header set (the public headers without `rocwmma/internal/`) into
`/usr/include`, where it shadows AMD's complete copy under
`/opt/rocm/include`. The first build died on `'internal/accessors.hpp'
file not found` from `/usr/include/rocwmma/rocwmma.hpp`, and **a `-I`
cannot fix it**: in HIP mode the host compilation pass searches the system
directories first. The package has no reverse dependencies; the script
removes it.
- **`--gcc-install-dir` must point at GCC 15.** Ubuntu 26.04 has
`/usr/include/c++/16`, and the ROCm clang picks the newest tree, whose
`<atomic_wait.h>` references `std::__detail::__notify_impl` at a symbol
version the installed `libstdc++.so.6` (GLIBCXX_3.4.35) does not export.
The link then fails with `undefined symbol`. Pinning the headers to the
GCC the runtime library came from fixes it.
Verify an installed runtime with `bash scripts/build-gufo.sh --verify`, or
`gufo diagnose` directly — it reports `[PASS]` plus the detected
architecture, driver and queue budget.
## Shape
### Payload
- **Model:** `ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`,
file `UD-Q8_K_XL` — 27B dense, 31,457,990,272 bytes, fetched and
sha256-verified (`9076f99a…`, the Hub LFS oid) by
`scripts/download-frank-dragoon-swift.sh`. It is an **abliterated
(uncensored)** fine-tune of Qwen3.8-27B. The repository also publishes an
`mmproj-BF16.gguf`, which is **not** used: this endpoint is text-only.
- **Licence:** the repo declares `other` — the **SWIFT OPEN LICENSE 1.0**,
which is *not* an OSI licence. That is a real constraint on redistribution,
noted here so nobody discovers it later. It is fine for a private fleet.
- **Replaced:** `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL`, still on disk and
still the rollback target for the payload.
The two targets are **structurally interchangeable for this engine**, which
is why the swap needed no engine change. Header-diffed directly:
| | unsloth | Swift-1.5 |
|---|---|---|
| `general.architecture` | `qwen35` | `qwen35` |
| tensors (count, names, order) | 866 | 866, **identical** |
| per-tensor `ggml_type` | Q8_0×453, F32×360, BF16×53 | **identical** |
| per-tensor dims | — | **0 mismatches** |
| MTP head | 4 tensors at `blk.64.nextn.*` | identical |
| geometry | 65 blk, 5120 embd, 17408 FFN, 24/4 heads | identical |
The two metadata differences that look alarming are both no-ops, and both
were confirmed against gufo's source rather than assumed:
- **`general.file_type` is 15 vs 7.** gufo's `gguf_reader.cpp` states it is
*"deliberately not consulted… The Unsloth UD-\*_K_XL artifacts this repo
targets carry a stale one — UD-Q8_K_XL records ftype 15 (Q4_K_M) … The
tensor table always is."* The file gufo already served lies about itself
here, so gufo reads the tensor table.
- **Swift adds `qwen35.attention.recurrent_layers`.** gufo never reads that
key anywhere in `src/`; layer kinds come from `full_attention_interval`,
which both files carry as `4` and which agrees with it.
### Serving shape
- **Context:** one session, full 262,144 tokens (the model's native window),
`SESSIONS=1`. The fleet's single stream is unchanged.
- **Spec method:** DFlash2, gufo's **adaptive** block policy
(`--draft-policy adaptive`, `--draft-tokens 7`). The drafter is the
**unchanged** z-lab `Qwen3.8-27B-DFlash2-Q4_K_M` (1.07 GiB) — it did not
move with the target, which is why `DRAFTER_DIR` is a separate tunable
from `MODEL_DIR` and why the deployer's stray-run sweep keys on the
drafter's filename rather than the target's.
- **Sampling:** greedy (`--temperature 0`) server-side, as every prior
frank-dragoon served. The harness overrides per request via `extraBody`
(1.0 / 0.95 / top-k 20).
- **Memory:** gufo's own table puts the 8-bit target at **50.58 GiB** peak
with context capacity 262,144, against the iGPU's ~114 GiB GTT aperture.
### Measured at the 2026-09-28 cutover
Live on Dankclaw. The **target** numbers are the Swift-1.5 load test
(a standalone run under the production flags, service stopped first); the
**engine** numbers are from the gufo cutover earlier the same day and are
unchanged by the payload swap.
- **9.2 s** `load_started` → `load_completed` for the Swift target, against a
20-minute `LLM_WAIT` budget. The unsloth target loaded in 13 s.
- **66.5 GiB** of the 116.7 GiB iGPU device at context capacity 262,144;
host RSS 30.6 GiB. No loader warnings, no skipped sections.
- Contract battery, all green on the Swift target: `/v1/models` →
`context_length 262144`, a chat completion carrying both `content` and
`reasoning_content`, a `tool_calls` response with
`finish_reason: tool_calls`, SSE streaming, and a **two-turn tool loop**
(tool call → tool result → follow-up) with prior reasoning preserved.
Arithmetic spot-check: "17*23" → `391`.
- Behavioural spot-check that this is the abliterated model: asked for a
two-sentence lockpicking scene, it wrote one without hedging or refusing.
- From the gufo cutover, on the unsloth target: a short-prompt live request
returned
`Server-Timing: ttft;dur=549.8, inter_token;dur=24.2` (≈41 tok/s).
Short-prompt decode is gufo's *winning* regime; see the depth table above
before quoting it as the endpoint's number.
### Tuning: the served flags are already optimal (2026-09-28)
Frank asked for iterative decode/prefill improvement at a 32k working point
without touching the model or the quantization. **Result: no win exists at
this engine's served settings, and nothing was changed.** Measured on real
repo prose and shell sources at **32,762 prompt tokens**, salted per run so
a warm prefix cache could not help, 3–5 repetitions each.
| lever | range swept | outcome |
|---|---|---|
| `--prefill-chunk` | 512 → 8192 | **no effect** — 410.7–412.4 t/s prefill across a 16× change |
| `--draft-tokens` | 5 → 10 | **flat** — 13.79–15.86 t/s decode, inside run-to-run noise |
| `--draft-policy` | fixed vs adaptive | 35.90 vs 35.50 (bench) — no effect |
| `--min-draft-tokens` | — | **rejected by the engine**: "DFlash2 requires … 1" |
| MTP backend | — | **unsupported for Q8**: `token_embd.weight` is Q8_0 |
| `--context` 262144 → 40960 | diagnostic | 13.93 vs 14.71 t/s — no gain, and it would break the declared client context |
The confirmation round at 5 reps **reversed** the 3-rep leaders: the pair
that looked best (`dt9 + chunk4096`) came last at 12.86 t/s, and the
**deployed configuration came first at 14.71 t/s**. Every difference sits
inside the noise band.
What *does* move the number is speculative decoding itself: **6.65 t/s off →
~15 t/s on, a 5.3× step** — and that is already switched on. Prefill is
insensitive to everything tried, which is consistent with gufo's own
`EXPERIMENTS.md`, where dozens of kernel-level alternatives are recorded as
rejected for "no release throughput improvement".
**Do not repeat this investigation without new information.** Two traps
cost most of the time, and both are in AGENTS.md's benchmarking policy:
`gufo bench` reports `acceptance=0.933 ± 0.000` on an identical trace hash
and **inverts** the draft-width ranking; and a live A/B needs ≥128 generated
tokens or `inter_token` is first-token noise.
## The template law: Qwen-Sharp is gone
`scripts/qwen-sharp-v22.4.0.jinja` is the fleet's chat template — it is
wired into `frank-archon` as `R9V_CHAT_TEMPLATE_FILE` and it was
`frank-dragoon`'s `--chat-template-file` under llama.cpp. **frank-dragoon can no
longer use it**, and the reason is stronger than "gufo lacks the flag":
> gufo has **no Jinja engine at all.** `src/models/qwen/chat_template.cpp`
> is 587 lines of hand-written C++ that appends strings to build ChatML.
> There is no `--chat-template-file`, no environment override, and no
> template body anywhere in the binary to point at. Qwen-Sharp is 453 lines
> of Jinja2 using `namespace()`, macros, `raise_exception` and image/video
> branches — supplying it would mean **reimplementing it in C++**, not
> handing gufo a file.
Given the choice on 2026-09-28, the decision was to **ship on gufo's stock
Jinja and accept the loss**. What that gives up, concretely:
| Qwen-Sharp feature | On gufo |
|---|---|
| `terse` system-prompt injection (the headline behaviour) | **gone** |
| in-band `<\|think_off\|>` / `<\|think_xhigh\|>` control | **gone** |
| `tool_call_format` `json` \| `xml` switch | **gone** |
| `max_tool_arg_chars` / `max_tool_response_chars` truncation | **gone** |
| `add_vision_id` → `Picture N:` prefixes | **gone** |
| effort instruction strings in the system prompt | kept (gufo has its own, same text) |
| `preserve_thinking` | kept (`--preserve-thinking on`) |
The `terse` injection is the one that matters most here, and it is worth
being blunt about the combination: this endpoint now runs an **abliterated**
model *without* the prompt that tells it to stop waffling. If verbosity
shows up, the cheapest fix is to send the `terse` text as an ordinary
**system message from the client** (`models.yml` / the harness) rather than
through a template — that recovers the behaviour without forking the
renderer. The full-fidelity options are reverting to llama.cpp or porting
Qwen-Sharp into the C++ renderer; both are worse trades today, and the
second is the same class of change that produced the 2026-09-21
"73,600-token prompt" incident.
## Client contract — what actually changed
Everything the fleet depended on still holds: port 7770, alias `frank-dragoon`,
`reasoning_content`, `tool_calls`, streaming SSE, and text-only (gufo
auto-discovers a `mmproj` beside the model; this payload has none).
The differences worth knowing:
- **No custom chat template — see
[The template law](#the-template-law-qwen-sharp-is-gone) above**, which is
the most consequential change on this endpoint. Thinking defaults to on at
`xhigh` with prior reasoning preserved (`--think auto
--preserve-thinking on`, spelled out in the serve script).
- **`reasoning_effort` accepts the full OpenAI set** (`none`, `minimal`,
`low`, `medium`, `high`, `xhigh`, `max`), all verified 200 against the
live endpoint. Qwen-Sharp rejected anything outside `{xhigh, medium, low}`
with an HTTP 500, which is why `models.yml` still carries a
`reasoningEffortMap` fold for this model. **The fold is kept on
purpose**: removing it would silently move every `high` role from `xhigh`
to `high` effort. Drop it as its own change.
- **Every field the harness sends is accepted.** `top_k`, `min_p`,
`presence_penalty`, `repetition_penalty` and `preserve_thinking` are all
outside gufo's documented request subset, and all returned 200. They are
applied from `models.yml`'s `extraBody` unchanged.
- **No KV-cache quantization flag.** gufo picks its own KV dtype; there is
no `--cache-type-k`. The old `CACHE_RAM=8192` tunable is gone with it.
Prompt reuse is on by default, and `--cache-disk` (restart-safe) is opt-in
and currently off.
- **Health/readiness split:** `GET /health` is process liveness and is what
the deployer gates on; `GET /ready` is model/backend readiness.
`GET /metrics` is Prometheus-format.
- **Long bodies:** `--max-request-bytes` defaults to 8 MiB.
## Failure modes worth knowing
- **A missing runtime is a deploy error, not a restart loop.** The handler
gates on `LLM_DRAGOON_RUNTIME` *before* it stops anything, and the serve
script exits with a `build-gufo.sh` pointer rather than exec'ing.
- **Host memory is the ceiling, and it is shared.** frank-dragoon and ComfyUI
both hold gfx1151 compute queues. gufo reads
`/sys/class/kfd/kfd/proc/*/queues/*/type` and caps itself
(`GPU_MAX_HW_QUEUES`, 2 resident for `serve llm`) so a full device
time-slices instead of failing; four HIP processes is the device ceiling.
- **Load is seconds, not minutes** (9.2 s measured cold on the Swift
target), so a redeploy no longer spends 20 minutes in `LLM_WAIT`.
- **A payload is two files in two directories.** The target and the drafter
no longer live together, so "the payload is present" is not one check; the
deployer gates on both paths and the serve script refuses to start if
either is absent. A symlinked "one payload directory" would hide that.
- **A v0.1.x runtime is a moving target.** Rebuilds are not automatic —
`RUNTIME_DIR` names a version, and the deployer will not silently
substitute another.
## Rollback
Two independent rollbacks, because two things changed on 2026-09-28:
1. **Payload only** — repoint `MODEL_FILE` in `scripts/frank-dragoon.env` at
`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/Qwen3.8-27B-UD-Q8_K_XL.gguf`
and redeploy. **The 29.3 GiB target was reclaimed on 2026-09-28**, so this
path now needs a re-fetch first: `scripts/download-frank-dragoon.sh`, then
the env edit. The DFlash2 drafter was never touched and still lives in
that directory.
2. **Engine** — [[frank-dragoon-unsloth-qwen38-27b-q8xl]]: the unsloth b11030
prebuilt is still at `/home/dank/GitPackages/unsloth-llama-b11030/`.
Revert `RUNTIME_DIR` and the gufo-only tunables, restore the serve
script's llama.cpp command, and redeploy. This is also the only way to
get `qwen-sharp-v22.4.0.jinja` back.
# Citations
- Runtime: https://github.com/gufo-org/gufo (pinned `v0.1.1`, MIT)
- Measured tables: `docs/models/qwen3.8-27b/BENCHMARKS.md` in that repo
- API contract: `docs/SERVER.md`; flags: `docs/CLI.md`; the C++ renderer
that rules out a Jinja template:
`src/models/qwen/chat_template.cpp` (587 lines)
- GGUF header evidence for the payload swap: both files' headers were parsed
and diffed directly (architecture, all 866 tensor names in order, per-tensor
`ggml_type` and dims) before either was loaded
- Build: `scripts/build-gufo.sh`; serve: `scripts/serve-frank-dragoon.sh`;
tunables: `scripts/frank-dragoon.env`
- Payload: `scripts/download-frank-dragoon-swift.sh` (target),
`scripts/download-frank-dragoon.sh` (DFlash2 drafter)
- Predecessor: [[frank-dragoon-unsloth-qwen38-27b-q8xl]]
serverAppConfigs-docs/LLM/frank-dragoon-halogen27b-dflash2.md
-200
View File
@@ -1,200 +0,0 @@
# frank-dragoon — halogen container, Qwen3.8-27B + DFlash2 (Dankclaw, ACTIVE 2026-10-05)
frank-dragoon is the Strix Halo iGPU endpoint on Dankclaw (`:7770`). Since
2026-10-05 it runs the **halogen** engine — `ghcr.io/peonist-ai/halogen` at a
pinned digest, launched by `scripts/serve-frank-dragoon.sh` under a root
systemd unit as a podman container — serving
**`peonist-ai/halogen-qwen3.8-27b`** (`qwen3.8-27b-p1w4d-d2.hgn`,
35,865,565,184 B ≈ 33.4 GiB, sha256 `274c3fc7…`, 1352 tensors) with the
**DFlash2** drafter at **one slot x 200,700 ctx**.
It replaced the ROCmFPX llama.cpp fork endpoint ([[frank-dragoon-rocmfpx-q6agent-uncensored-mtp]]),
which remains the rollback target with its payload on disk.
## Why swap
|Metric|ROCmFPX Q6_0_ROCMFPX_AGENT + MTP (retired)|halogen qwen3.8-27b + DFlash2 (active)|
|---|---|---|
|Prose decode|27.6–28.1 t/s measured (4.8k depth, acceptance 0.868)|31.7 t/s mean across eight text workloads (publisher card, bf16 reference hardware)|
|Copy-echo decode|31.4 t/s|publisher reports echo inside the same 31.7 mean|
|Context|1 x 262,144|1 x 200,700|
|Format|GGUF (fork-private tensor types 102/103)|`.hgn` — loads in NO other engine|
The swap was accepted with eyes open on two product regressions, both
confirmed by Frank before deploy:
1. **Text-only.** halogen loads the language tower only; README "honest
limits": *"The vision encoder is ignored; image inputs are not supported."*
The mmproj-era vision role moves wholly to frank-zealot.
2. **Censored base.** The checkpoint is a finetune of the censored
`Qwen/Qwen3.8-27B` base, built to make its DFlash2 head strong. The
ROCmFPX AGENT file was an *Uncensored* build; refusals can return.
## Shape and why
`scripts/frank-dragoon.env` carries every tunable with its evidence; the load-
bearing numbers:
- **`HALOGEN_KV_SLOTS=1` + `HALOGEN_DRAFTER=2`** — the engine offers batch OR
speculation, never both (FLAGS.md: "Batching and speculation are exclusive:
a step batches when more than one conversation is running, so the drafter
stays off with more than one slot"). Slots>1 silently loses DFlash2; the
sibling halogen engine measured 41 -> 14 t/s when slots went up. frank-dragoon
was a one-conversation endpoint before (PARALLEL=1), so the contract holds.
- **`HALOGEN_SLOT_CTX=200700`** — the per-conversation HARD ceiling: a longer
prompt is a 400 naming the limit, never a truncation. The harness side is
wired to it: `models.yml` `contextWindow: 200700`, and OMP compacts at 70%
= 140,490, leaving 60,210 of the slot for the reply.
- **`HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200`** — coupled
pair (FLAGS.md table): a request at the cap must not outlive the timeout.
65536 > the harness's 57,344 `maxTokens` so an eager request never meets
the engine's 400.
- **KV math:** 1 slot x 200,700 x 64 KiB/token (d2 KV = 2x fp8) = 12.2 GiB
pool, inside the ~65 GiB left after 33.4 GiB mmap'd weights + engine working
set + host baseline on the 121.2 GiB box. The engine sizes the pool itself
and **prints a boot ledger naming any clamp** — the post-deploy check reads
that ledger, not a formula.
- **`HALOGEN_CACHE_MB=16384` (pinned 2026-10-06)** — auto is OFF on this
box: the floor is 2 x a full-context entry (36.9 GB at 262k; one entry
is ~13.7 GB at our 200,700-slot x ~70 KB/token) and the ledger had
35.7 GB free, so the engine refused and every deep agent request
re-prefilled its whole 79-81k context — 84.9-179.8 s measured
(serve_api lines, 2026-10-06) for ~500-650 new tokens per turn. The
README prices the cache at ~20x time-to-first-token on a warm turn;
entries evict LRU by bytes; the reserve (HALOGEN_CACHE_RESERVE_MB,
default 8192) is re-checked against live MemAvailable on every store.
- **No server-side sampler:** an absent temperature is GREEDY. The card's
thinking recipe (temp 1.0 / top_p 0.95) lives in `models.yml` `extraBody`
— on the wire only. The template defaults `reasoning_effort` to **xhigh**
when a request sends none, and **thinking tokens consume the
`max_tokens` budget**: a budget exhausted mid-thought returns
`finish_reason: "length"` with EMPTY `content` and the partial reasoning
in `reasoning_content` — clients must read both fields.
## Client contract deltas vs llama.cpp
|Behavior|halogen|
|---|---|
|Prompt over slot|400 naming the limit (no truncation)|
|`max_tokens` over cap|400 naming the limit (never silent truncation)|
|full-length request outliving `HALOGEN_QUEUE_TIMEOUT`|503s for everyone queued|
|`timings`|every response carries llama-server-shape `timings` (`draft_n`, `draft_n_accepted`) — the drafter check reads the RESPONSE, and it proves speculation fired|
|`response_format`|NOT supported (no constrained decoding)|
|thinking field|`reasoning_content` (same as before)|
|`reasoning_effort`|same Qwen3.8 template; outside {xhigh,medium,low} rejected — models.yml fold stays; unset defaults to xhigh, `none` disables thinking|
|tools|server-side tool calls advertised — round-trip MUST be re-probed after deploy|
|image input|ignored/error — route to frank-zealot|
## Deploy mechanics
- **Image:** `ghcr.io/peonist-ai/halogen@sha256:dc0a39a0016d6cfc58a197978febaafdf8d28403f724ded6111d98b5fb7ac0ea`
(published tag 0.1.4). The deployer gates the exact digest exists in podman
storage before dispatch. NOT the 2026-09 `halogen-flash-server` repo — that
was the Flash-Next w4b era, different image, retired.
- **Payload:** `scripts/download-frank-dragoon-halogen.sh` (fetch law: `curl -f`,
exact byte count, LFS sha256 where the Hub publishes one, `--verify-only`,
token never in argv) into `/home/dank/models/halogen-qwen3.8-27b/`. The
tokenizer is a FLAT DIRECTORY (`tokenizer/`, 5 files — tokenizer.json,
vocab.json, merges.txt, chat_template.jinja, tokenizer_config.json),
mounted whole at `/tokenizer`; the engine's `HALOGEN_TOKENIZER` default is
that directory, so the container gets no tokenizer flag (model-card
docker-run parity). The deployer and serve script refuse a missing file or
ANY symlink: an HF cache snapshot symlinks each entry into `blobs/` and
those links dangle inside the container mount.
- **Unit:** `scripts/frank-dragoon.service` (root, `User=dank`, relinked into
`/etc/systemd/system/`), `ExecStart` = serve script, `ExecStop` = `podman
stop -t 120 frank-dragoon-halogen`, script runs `podman run` FOREGROUND
(2026-10-06: the 2026-10-05 `-d` + `exec podman wait` shape captured
only the container ID — the engine ledger existed nowhere on disk and
the post-deploy grep below read a one-line hex file; foreground run
returns the container's exit code so `Restart=always` is real and the
ledger lands in the log). Engine stdout goes to
`/home/dank/logs/frank_dragoon.log` (the boot ledger lives there, not in
the journal).
- **Cutover:** `deploy_llm_dragoon` in `scripts/deploy-remote.sh` sweeps the
legacy bare `llama-server --model /home/dank/models/kingjones-*` process and
the legacy `frank-dragoon` container name by NAME (never `pkill -f
llama-server` — that hits frank-zealot), retires the old quadlet if the 2026-09
halogen era's unit lingers, waits port 7770 free, relinks, health-gates.
- **Bench law:** stop+mask the unit before any GPU experiment on this box
(AGENTS.md); `/health` answers 503 while the 33.4 GiB payload loads — that
is loading, not failure.
## Verification (post-deploy checklist)
1. `grep -iE 'clamp|kv|pool|drafter|dflash|cache' /home/dank/logs/frank_dragoon.log`
— ledger shows the pool at the asked shape, DFlash2 active, the
prompt cache line naming the pinned budget, NO clamp line. EXPECT the
WARNING "smaller than ONE full-context entry (18.4 GB at
kMaxCtx=262144)": it sizes against the NATIVE window, not our
200,700 slot — the deepest possible entry (~14 GB at ~70 KB/token)
fits the 16 GB pin, and the 262k entry it fears cannot exist (over
slot is a 400). Probe-proven 2026-10-06: the warm turn hit with
cache_n=16,384.
2. `curl -s http://192.168.0.143:7770/health` -> 200.
3. Real completion with `reasoning_effort` + temp/top_p from the wire recipe;
`reasoning_content` populated; decode rate from the timings.
4. Spec decode fires: multi-token step behavior visible in rate; compare
against DRAFTER=0 arm if in doubt (never against gufo — bench law).
5. Harness round-trip: an OMP call through `llamacpp (dankclaw)/frank-dragoon`
incl. a tool call (V1TOOLS).
6. Prompt > 200,700 tokens returns the 400 naming the limit (contract test).
## Tuning notes (2026-10-06)
Measured on real agent traffic (serve_api lines) the day after cutover:
- **Decode is depth-bound, and the KV path is the reason.** Shallow
prompts (< 1k tokens) decode 30-50 t/s at commit 4-6.3/round; 46k runs
5.5, 63-67k runs 4.1-6.5, 79-81k runs 3.3-3.8. The round-time slope is
~8 us/token of context: the KV pool reads at ~8 GB/s effective while
the weights stream ~180-240 GB/s (the engine's README puts batch-1
weight streaming at the 240 GB/s hardware wall). Commit also falls
from 4-6 to 2.5-3 accepted/round as the conversation grows.
- **No supported flag touches decode.** `docs/FLAGS.md`: HALOGEN_W4A4 /
W4A4_EXCL are prefill-only ("decode is unaffected either way"); the
undocumented kernel levers are declared unsupported and their defaults
are the measured winners. 0.1.4 is the latest release (CHANGELOG
head; kernels unchanged since 0.1.0). The publisher's tg bench never
measured depth — its tg rows run over the ten SHORT eval prompt
shapes — so the README's "decode barely degrades at depth" claim was
never exercised at 46-80k.
- **A/B'd the one FA lever with public precedent (2026-10-06 stop-window
probe): rejected.** Issue #12's `HALOGEN_FA_TQ=256` (the 64 GB-carve
crash workaround) vs the default tile — same 80k filler prompt,
128-token essays, the runtime's own `timings`: decode 6.35 vs 6.34 t/s
cold and 5.74 vs 5.76 warm (±0.3 % = noise; draft acceptance identical,
53/74 and 58/69), while prefill LOST 9.5 % on this topology (447.5 ->
404.8 t/s). The depth collapse is not an FA tile artifact — it sits
deeper than the knob. The engine binary advertises 40+ undocumented
`HALOGEN_*` kernel levers (FA/DN/MATMUL families, readable via
`grep -aoE "HALOGEN_[A-Z0-9_]+"` on the binary); `FA_TQ` is the only
one with public evidence, and it moved nothing that matters here.
- **Routing is the applied mitigation (2026-10-06):** the harness's
deep-conversation roles (`slow`, `advisor` in `~/.omp/agent/config.yml`)
moved to frank-archon (radiance on Dankserver: fp8 KV at
7.32 KiB/token = 8.7x less KV to read per round at depth, MTP depth 3,
200k window, prefix-cache host+disk tiers). frank-dragoon keeps OpenWebUI
and shallow bursts, where it runs 30-50 t/s. frank-archon's
depth-decode rate is designed-for but not yet measured on record.
- **The lever that existed was the prompt cache, and it was OFF** (see
the shape bullet). The pin turns the 85-180 s re-prefill per deep
turn into a resume; decode itself is expected UNCHANGED by the pin —
the cache resumes prefill, it does not touch the round loop.
Probe-proven (two synthetic 17.3k turns, 2026-10-06): cold prefill
27.0 s (640 t/s, cache_n 0) -> warm turn cache_n 16,384, prompt_n 946
(the extension + one 2048-aligned chunk), prefill 1.7 s — 16x on the
pair, answer correct warm.
- **Everything else stays put.** dflash2 over mtp (publisher 31.71 vs
26.88 mean; at depth both pay the same KV term and dflash2's higher
commit wins), KV_SLOTS=1 (batch XOR speculation), SLOT_CTX=200700
(Frank's window), W4A4 shipped (+9 % prefill, decode unharmed).
## Rollback
Revert the frank-dragoon swap commit and redeploy `dankclaw --service frank-dragoon`:
the ROCmFPX payload (`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/`),
the fork build (`/home/dank/GitPackages/kingjones-ROCmFPX/build`), and its
`scripts/build-rocmfpx.sh` pin (commit `36e9acd`, seam-tested) all remain on
disk. The halogen payload can stay: it costs nothing while the unit points
elsewhere.
serverAppConfigs-docs/LLM/frank-dragoon-ik-uncensored.md
-149
View File
@@ -1,149 +0,0 @@
---
type: reference
description: frank-dragoon on Dankclaw serving the mradermacher Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M GGUF via ik_llama.cpp on the RTX 3090 + Strix Halo hybrid — measured performance envelope, root-cause analysis, tuning levers, and traps
tags: [llm, frank-dragoon, ik_llama, dankclaw, rtx3090, strix-halo, performance]
timestamp: 2026-09-16T03:30:00-07:00
---
# frank-dragoon — ik_llama.cpp, Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M
**RETIRED 2026-09-16, after one day.** Replaced by [[halogen-flash-server]],
which was the runtime before it as well — the fleet went ik → halogen → back.
Its unit, env, serve script and the Qwen-Sharp template are archived in the repo
under `scripts/.AGENTTRASH/*.ik-llama.cpp` and `*.ik-llama-qwen-sharp`, and the
payload, build and chat template stay on disk. The measurements below are not a
stale plan: they are the complete account of what this model costs on this
hardware, and they are why the fleet went back. It served
`mradermacher/Qwen3.8-Flash-Next-Uncensored` (111 GiB GGUF + mmproj-f16) at
`192.168.0.143:7770` while it ran.
## Shape
- Runtime: ik_llama.cpp master `d85a1d7`, built at
`/home/dank/GitPackages/ik_llama.cpp/build` with `-DGGML_CUDA=ON
-DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_AVX512=ON -DGGML_AVX512_VBMI=ON
-DGGML_AVX512_VNNI=ON -DGGML_AVX512_BF16=ON -DLLAMA_CURL=OFF`.
- Unit: **root systemd unit** `/etc/systemd/system/frank-dragoon.service`
symlinked to `scripts/frank-dragoon.service` in the repo; enabled at boot;
`Restart=always`. Serve script `scripts/serve-frank-dragoon.sh` + env
`scripts/frank-dragoon.env`.
- Topology: 3090 (PCIe x4 in the M.2 dock, link downgraded from x16) carries
dense/attention/KV (~11.5 GiB); ALL MoE expert tensors (~84 GiB) and the
~33 GiB PLE n-gram table stay in LPDDR via CPU buffers.
- Client contract: greedy default (`--temp 0.0`), reasoning on (deepseek
format, unlimited budget), vision via mmproj, slot-state persistence under
`scripts/slot-state`, 262144 ctx, 1 slot.
## Measured performance envelope (2026-09-15/16 campaign, all single-instance verified)
| config | decode tok/s | prefill tok/s | verdict |
|---|---|---|---|
| shipped (T16, `-ot exps=CPU`, NO_PINNED_WEIGHTS) | **3.8-4.0** | ~193 at 1.8k / 191 at 32k | shipped |
| T8 / T4 / T2 / T1 | 4.18 / 4.20 / 4.26 / 3.97 | small-prompt only | decode flat across threads |
| `-ncmoe 42` (6 layers' experts on GPU) | 4.54 | 32 (small prompt) | VRAM 22.0/24.6 GiB — fragile |
| `-ncmoe 48 -thp` | 4.60 | 18 | **110 GiB anon host buffer — co-tenant killer, never ship** |
| `--prefetch-experts` | 4.03 | — | no effect (page cache already resident) |
| `--defer-experts --defer-ple -thp` | crashes | — | experts under `-ngl 999` still target CUDA0 → cudaMalloc OOM |
| mainline llama.cpp `37b53fd` (`--n-cpu-moe 48`) | **1.01** | 34 | ruled out — CPU MoE kernels 4x slower than ik's iqk |
- Large-prompt prefill ≈ 192 tok/s is the **CPU AVX512-VNNI compute ceiling**
for CPU-resident experts (expert weight reads amortize across the batch;
compute-bound). Halogen's "4x llama.cpp family" prefill claim is therefore
unreachable with experts on CPU — the physics caps hybrid MoE prefill here.
- Decode is gated by a **serial per-token critical path**: ~100 CPU↔GPU
backend splits per token (`graph splits = 100` in the log); the 3090 sits at
**5% SM utilization** during decode; thread count does not matter (T1 ≈
T16 ≈ 4 tok/s). The GPU sections are launch/sync-bound, not compute-bound.
## Root cause chain (perf evidence)
1. `perf` at T16: ~53% cycles in `ggml_graph_compute_thread` + ~37% in
`ggml_compute_forward_mul_mat_id`, of which **~98% is the `pause`
instruction** (thread-pool barrier spin). Actual vec_dot math: ~4%.
2. T1 = T16 decode (3.97 vs 3.8-4.0) ⇒ the barrier spin is threads waiting on
a serial critical path, not parallel overhead.
3. GPU 95% idle during decode ⇒ the critical path is the GPU-section
launch/sync overhead per split (x4 dock link, eager kernel launches).
4. ik's CUDA-graph capture would collapse those launches but **aborts** on
qwen4exp: the per-layer hyper-connection broadcasts (`ggml_repeat_4d`,
4 sites in `src/graphs/build_qwen4exp.cpp`) resolve to a REPEAT node whose
CUDA path does a capture-illegal operation
(`CUDA error: operation not permitted when stream is capturing` → abort).
5. Memory placement is NOT the lever: mmap-off/anon/THP/deferred variants all
land at 3.8-4.6 tok/s.
## Why the 30 tok/s halogen bar is out of reach (honest record)
Halogen's engine held the whole 115.55 GiB w4b checkpoint resident in LPDDR
(`HALOGEN_FLASH_PIN_TRUNK=1`) and served from one engine. This deployment must
split 84 GiB of expert tensors across a 24 GB GPU + CPU, and every public
runtime pays the split penalty above. Paths to >30 tok/s, in order of
realism: (a) a runtime with capture-safe CUDA graphs + dedicated hc ops —
mainline is building exactly this (ggml-org PR #28901 "qwen4exp: add hc ops");
re-evaluate mainline when its hc-ops + graph handling mature; (b) experts on
GPU — impossible, 84 GiB > 24 GiB; (c) a smaller model.
## Upstream issues tracking the slow TG path
ik_llama.cpp: #2374 (QSA opt POC), #2406 (single-threaded sigmoid), #2433
(cache-miss full reprocess), #2460 (PLE history correctness). MTP/speculative
decoding is unavailable for this GGUF — mradermacher stripped the NextN head
(verified: no `blk.48.*`/nextn tensors in the file).
## Traps (learned the hard way 2026-09-15/16)
- **Pinned host buffers strand the host**: a default-policy start
page-locks ~108 GiB (`CUDA_Host buffer`), evicting co-tenants into swap and
starving sshd until a power cycle. `GGML_CUDA_NO_PINNED_WEIGHTS=1` + the
serve-script guard exist for this. `-ncmoe N` with default NO_PINNED
settings produces the same 108 GiB pinned buffer — never run it without the
guard.
- **Any `-ot`/`-ncmoe` override disables mmap** for the whole model load
(`create_tensors` returns false → `ml.use_mmap = false`): the "mmap-backed
CPU buffer" comment in the serve script is wrong on the mmap word; the
buffer is anonymous. All placement variants perform alike anyway.
- **`--defer-experts` does not keep experts off the GPU** under
`-ngl 999` — only PLE defers (32.78 GiB). With ngl 999 the load dies with
`cudaMalloc failed: out of memory` at ~79.7 GiB on device 0.
- **The systemd unit self-heals through manual tests**: `pkill` the server and
the unit respawns it ~4 s later (Restart=always) — two instances, port
races, phantom measurements. For manual experiments:
`sudo systemctl stop frank-dragoon.service` first (it is a root unit; plain
`systemctl --user` lines in older docs are stale), and restore with
`sudo systemctl start frank-dragoon.service`.
- Verify "HTTP server listening" in the CURRENT log and
`pgrep -xc llama-server` == 1 before trusting any measurement.
- FA must stay ON — FA-off crashes the loader on this arch.
- Do not run builds concurrently with decode measurements (`-j16` starves the
service and corrupts both results).
## Rollback
Stop and disable the root unit, trash it, restore the halogen quadlet from
`~/.AGENTTRASH/20260915-ik-swap/halogen-quadlet` to
`~/.config/containers/systemd/`, `systemctl --user daemon-reload`, start;
the .hgn checkpoint and the kingjones STRIX llama.cpp payload remain on disk.
## Hybrid campaign envelope (2026-09-16, 3090 + Strix Halo)
Why this deployment could not be tuned into a production endpoint: decode
3.8–4.3 tok/s flat across thread counts (T1 ≈ T16); prefill ~192 tok/s at
1.8k–32k prompts = the **CPU AVX512-VNNI compute ceiling** for CPU experts.
The decode root cause is structural: ~100 CPU↔GPU backend splits per token
with the GPU at 5 % SM during decode — launch/sync-bound over the PCIe x4
dock, not compute- or memory-bound. T16 profiling shows ~90 % of cycles in
`pause` (ggml_barrier spin): threads wait on the serial critical path.
Memory placement (mmap/anon/THP/deferred) does not matter — all variants
landed 3.8–4.6.
- CUDA-graph capture aborts: qwen4exp's hyper-connection broadcasts use
per-layer `ggml_repeat_4d`, whose CUDA path is capture-illegal; until ik
gains capture-safe repeat/hc ops (mainline PR #28901 direction), graphs
stay dead.
- Mainline llama.cpp itself runs 1.0 tok/s on this model (CPU MoE kernels
4× slower than ik's iqk) — do not switch engines for speed.
- MTP/speculative decoding is impossible for mradermacher i1 GGUFs: the
NextN head is stripped (no `blk.48` tensors).
- Tuning lever ranked first: `-ncmoe 42` (6 layers' experts on the 3090) →
decode 4.54, small-prompt prefill 32 — but VRAM 22/24.6 GiB at 262k ctx,
an operator OOM-margin decision, and it needs `GGML_CUDA_NO_PINNED_WEIGHTS=1`.
serverAppConfigs-docs/LLM/frank-dragoon-kairic-edge-qwen38-27b.md
-48
View File
@@ -1,48 +0,0 @@
---
type: reference
title: "Kairic Edge Qwen3.8-27B IU4 (W4A4) on Dankclaw (frank-dragoon)"
description: "The RETIRED frank-dragoon runbook (2026-09-21, same-day swap; replaced by [[frank-dragoon-unsloth-qwen38-27b-q8xl]]) — the Kairic Edge runtime (ciru-ai/ROCmFPX, branch kairic-edge-qwen38-27b-v1.2, commit 205a3e5, TheRock 7.15.0, Composable Kernel) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (27B dense, IU4 W4A4) on the Dankclaw Strix Halo iGPU: the source-built runtime, the full-window/one-stream shape, the MTP depth 4 + n-gram spec method, the three PromptForge sidecars, the compatibility mode, the payload, the client contract, and the failure modes worth knowing."
tags:
- llm
- kairic-edge
- rocmfpx
- dankclaw
- frank-dragoon
- strix-halo
- gfx1151
- iu4
- w4a4
- rocm
- mtp
- promptforge
timestamp: 2026-09-21
---
# frank-dragoon — Kairic Edge Qwen3.8-27B IU4 (W4A4) (RETIRED)
> **RETIRED 2026-09-21 (same-day swap).** This was the ACTIVE frank-dragoon
> runbook on 2026-09-21 before it was replaced by
> [[frank-dragoon-unsloth-qwen38-27b-q8xl]] later the same day. The payload
> and runtime remain on disk as the rollback target.
## What it served
- **Runtime:** Kairic Edge (ciru-ai/ROCmFPX, branch
`kairic-edge-qwen38-27b-v1.2`, commit 205a3e5, TheRock 7.15.0,
Composable Kernel), root systemd unit, port 7770.
- **Model:** `jcbtc/Qwen3.8-27B-IU4-Kairic-Edge` (a 27B dense model, IU4
W4A4) — main GGUF + FFN/GDN/GDN-Output `.pfs` sidecars.
- **Shape:** one 262,144-token conversation (single stream), native MTP
depth 4 + n-gram drafting, compatibility mode.
- **Memory:** the 114 GiB GTT aperture on the Strix Halo iGPU.
## Rollback
Payload and runtime intact on disk. See [[frank-dragoon-unsloth-qwen38-27b-q8xl]]
for the current ACTIVE runbook and [[frank-dragoon-unsloth-qwen38-flash-next]]
for the predecessor.
# Citations
- Kairic Edge runtime: https://github.com/ciru-ai/ROCmFPX
- Model: https://huggingface.co/jcbtc/Qwen3.8-27B-IU4-Kairic-Edge
serverAppConfigs-docs/LLM/frank-dragoon-rocmfpx-q6agent-uncensored-mtp.md
-177
View File
@@ -1,177 +0,0 @@
---
type: reference
title: "frank-dragoon on Dankclaw — ROCmFPX llama.cpp, Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP"
description: "The active frank-dragoon runbook — the kingjones30/ROCmFPX llama.cpp fork (pinned commit, gfx1151) serving kingjones777's Uncensored Q6_0_ROCMFPX_AGENT GGUF (ftype 114, protected output heads), with the repo's own MTP-Q4_0 head drafting four tokens at one native 262,144 window, f16 KV, the BF16 projector for vision, and the measured rejection of the ngram-mod chain."
tags:
- llm
- frank-dragoon
- dankclaw
- llama-cpp
- rocmfpx
- gfx1151
- gguf
- spec-decode
- mtp
timestamp: 2026-10-03
---
# frank-dragoon — ROCmFPX Uncensored Q6_0_ROCMFPX_AGENT + MTP (gfx1151)
> **RETIRED 2026-10-03 → 2026-10-05 (rollback target; payload, fork build and
> build script all on disk).** Replaced by [[frank-dragoon-halogen27b-dflash2]]
> (DFlash2, publisher 31.7 t/s mean vs the 27.6–28.1 measured here). This page
> remains the contract record for the rollback. Original banner follows.
> ACTIVE since 2026-10-03. Cut over from
> [[frank-dragoon-rocmfpx-q8agent-mtp]] (ALIGNED Q8_0_ROCMFPX_AGENT, 17.0 t/s)
> on Frank's request: the *uncensored* variant of the same protected-heads
> family, one quant rung down. The swap turned out to be a strict upgrade on
> the axis that matters here: **decode 27.6–28.1 t/s at a 4,813-token prompt
> against 17.0 at a 2,202-token prompt** — faster with a deeper context, not a
> same-depth comparison, but the sign is not in question.
## Identity
| | |
|---|---|
| Host / device | Dankclaw (192.168.0.143), Radeon 8060S iGPU `gfx1151`, ~114 GiB GTT aperture |
| Endpoint | `http://192.168.0.143:7770` — alias `frank-dragoon`, `/v1` OpenAI surface |
| Runtime | **`kingjones30/ROCmFPX`** llama.cpp fork @ `36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built by [`scripts/build-rocmfpx.sh`](../../scripts/build-rocmfpx.sh) — unchanged from the Q8 era; the fork already knew type 102 |
| Unit | root systemd `frank-dragoon.service` ([`scripts/frank-dragoon.service`](../../scripts/frank-dragoon.service)), `User=dank`, `Restart=always` |
| Serve script | [`scripts/serve-frank-dragoon.sh`](../../scripts/serve-frank-dragoon.sh) (owns the argv), tunables [`scripts/frank-dragoon.env`](../../scripts/frank-dragoon.env) |
| Weights | `/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf` — 24,856,607,008 B, sha256 `e966deb3…` |
| Draft | `…/mtp-Qwen3.8-27B-Q4_0.gguf` — 1,680,271,648 B, sha256 `051a1764…` (byte-identical to the aligned repo's head; hardlinked, not re-downloaded) |
| Projector | `…/mmproj-Qwen3.8-27B-BF16.gguf` — 931,145,888 B (hardlinked from the aligned payload dir; the same file the Q8 era served) |
| Fetch | [`scripts/download-frank-dragoon-kingjones-uncensored.sh`](../../scripts/download-frank-dragoon-kingjones-uncensored.sh) — hardlinks the draft/projector from the aligned dir when present, otherwise fetches; `-f`, byte-count + LFS sha256 verified, `--verify-only`, token never on the command line |
| Seam | `tests/frank_dragoon_unit.bats` (payload gates, deployer/serve parity, spec block, KV dtype) |
## Why this file
`kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF` ships the same
"protected heads" construction as the aligned repo, in the uncensored tuning.
Parsed header of the deployed file:
| Field | Value |
|---|---|
| ftype | **114** — the fork's 64-bit-block family, ≈7.44 bpw |
| Tensor mix | 353 F32 + **339 type-102 (`GGML_TYPE_Q6_0_ROCMFPX`)** + **159 type-103 (`Q8_0_ROCMFPX`)** |
| Protected heads | 64 output-head tensors kept at plain precision — the AGENT construction from the aligned repo, at the 6-bit rung |
| Arch | qwen35, 64 layers, `full_attention_interval 4`, `context_length 262144` |
| Embedded sampling recipe | temp 1.0 / top_p 0.95 / top_k 20 |
The same lock-in law as the Q8 era applies: types 102 and 103 exist in no
loader but this fork; stock llama.cpp and gufo refuse the file at load
(`unknown tensor type`). The loader was not re-litigated — the file picks the
runtime, and the pin in `scripts/build-rocmfpx.sh` (commit by SHA) stays.
Rejected sibling: `Q6_0_ROCMFPX_LEAN` (6.45 bpw, no protected heads) — the
protection is what keeps the output-head logits at full precision and the
drafter accurate, so it stays deployed.
## Shape and budget
Unchanged from the Q8 era — the swap was payload-only:
| Dial | Value | Why |
|---|---|---|
| `--ctx-size` | 262,144 | the native window, no rope scaling |
| `--parallel` | 1 | one conversation, all memory to it |
| KV dtype | **f16** (both K and V) | ~64 KiB/token; free inside a 114 GiB aperture |
| `--fit off` | on | the auto-fitter would silently resize the window |
| `--flash-attn on` | on | required for the long-window attention path |
| `--direct-io`, `--ctx-checkpoints 16`, `--cache-ram 8192`, `--cache-idle-slots`, `--jinja`, `--reasoning on --reasoning-format deepseek --reasoning-budget -1` | | same as [[frank-dragoon-rocmfpx-q8agent-mtp]] |
Spec decode: `--spec-type draft-mtp`, the repo's `mtp-Qwen3.8-27B-Q4_0.gguf`,
`--spec-draft-n-max 4`, pinned `--spec-draft-device ROCm0`.
Measured steady state: GTT **~61 GiB** under live load (host baseline 16.7),
boot to `/health` **~15 s**.
## Measured on this payload (2026-10-03, production service, `/completion` timings, temp 0, `cache_prompt: false`)
| Traffic | Decode t/s | Prefill t/s | Draft acceptance |
|---|---|---|---|
| Prose, 4,813-token prompt, 512 tokens | **27.57 / 28.10** (two runs) | ~205–217 | **396/456 = 0.868** → mean acceptance length ≈ 4.4, i.e. drafting at nearly the full n-max 4 |
| Schema echo (structured repetition) | **31.44** | — | 639/639 = 1.000 |
| Block copy (text repeated verbatim) | ~28.0 | — | 0.848 |
vs the Q8 era's 0.462 acceptance at a 2,202-token prompt: the MTP head drafts
materially better on this file. The card's claimed 0.953-class acceptance is
no longer science fiction on real prose — but keep reading it as a file
property, not a throughput promise.
## The ngram-mod chain — measured and rejected
Frank's ask: layer `--spec-type draft-mtp,ngram-mod` on top of MTP for an
extra gain. Measured in three bench windows (production service stopped per
the benchmarking law; artifacts `/home/dank/fl-ngram-ab/` on Dankclaw),
comparing `draft-mtp` alone vs the chain on the same traffic:
| Traffic | MTP alone | MTP + ngram-mod | Delta |
|---|---|---|---|
| Prose | 27.57 t/s | 23.40 t/s | **−15 %** |
| Block copy | 28.0 t/s | 26.1 t/s | **−7 %** (with ngram-mod demonstrably firing: 138 proposals/run) |
Mechanism, straight from arm B's boot log:
```
recurrent state rollback is not compatible with 'ngram-mod' - disabling rollback support
```
On the qwen35 hybrid (Gated DeltaNet recurrent layers) adding the ngram layer
disables the rollback path MTP's own verification relies on, so acceptance —
and throughput — drop on *all* traffic, even where the ngram matcher fires.
**Deployed decision: `draft-mtp` alone.** Chaining is not free on recurrent
models; the fleet law in `AGENTS.md → llama.cpp serving` records it.
## Bench-window hardening laws (learned building the A/B)
1. **Health 200 is not proof of ownership.** After boot, verify the pid you
started owns the port (`ss -ltnp`); a zombie server from a prior window
answers `/health` too and silently serves your arm.
2. **Kill scratch servers with SIGKILL after a grace period.** llama-server's
graceful SIGTERM waits on open keep-alive connections — a harness holding
the socket leaves the process (and its ~24 GiB of GTT) alive after the
script exits.
3. **Prove the port is free and GTT dropped between arms**, or the second
arm measures a collision.
## Client contract
Unchanged from the Q8 era and re-verified on this payload 2026-10-03: model id
`frank-dragoon`, native tools, thinking in `reasoning_content`,
`reasoning_effort` ∈ {xhigh, medium, low} only (anything else HTTP 500),
card-recipe sampling defaults server-side, vision through the BF16 projector
(red PNG → **`Red`**, 74 prompt / 165 completion tokens). The traps list —
`content: null` when reasoning eats the budget, the self-recovering MTP
boundary log line, `enable_thinking:false` not `reasoning_effort:"none"` — is
in [[frank-dragoon-rocmfpx-q8agent-mtp]] and still applies.
Sweep key for scratch servers is now the payload name:
`pkill -f 'Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT'` — never `pkill -f
llama-server` (frank-zealot is also one).
## Operations
```bash
./scripts/deploy.sh dankclaw --service frank-dragoon
systemctl status frank-dragoon
journalctl -u frank-dragoon -n 200 --no-pager
scripts/download-frank-dragoon-kingjones-uncensored.sh --verify-only
```
## Rollback
| Target | How |
|---|---|
| **ALIGNED Q8_0_ROCMFPX_AGENT** (previous shape, 17 t/s) | the payload dir `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/` is untouched on disk; revert the swap commit (unit/env/serve trio) and redeploy, or point `FRANK_DRAGOON_MODEL_FILE` + `LLM_DRAGOON_MODEL_FILE` at the old paths. Verify with `scripts/download-frank-dragoon-kingjones.sh --verify-only` |
| gufo Q4_K_XL era | git tag `frank-dragoon-gufo-q4kxl-20261001`, **but the gufo payloads (UD-Q4_K_XL 16.35 GiB, UD-Q8_K_XL + DFlash2 drafter) were removed from disk 2026-10-03 with Frank's approval** — this rollback now requires re-download per [[frank-dragoon-gufo-qwen38-27b]] |
## When it misbehaves
Same decision tree as [[frank-dragoon-rocmfpx-q8agent-mtp]]:
`unknown tensor type` ⇒ off the pinned fork; SIGSEGV in
`ggml_compute_forward_set_rows` ⇒ pre-`36e9acd` build; slow decode with no
error ⇒ draft not engaged (`draft_n` in `/completion` timings) or an orphaned
second copy eating GTT (`/sys/class/drm/card1/device/mem_info_gtt_used`);
500 naming Jinja ⇒ `--jinja` lost; 500 naming the template ⇒ unmapped
`reasoning_effort` in `models.yml`.
serverAppConfigs-docs/LLM/frank-dragoon-rocmfpx-q8agent-mtp.md
-232
View File
@@ -1,232 +0,0 @@
---
type: reference
title: "frank-dragoon on Dankclaw — ROCmFPX llama.cpp, Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (retired)"
description: "The retired frank-dragoon runbook — the kingjones30/ROCmFPX llama.cpp fork (pinned commit, gfx1151) serving kingjones777's Q8_0_ROCMFPX_AGENT GGUF, a ftype-115 file no other loader can read, with the repo's own MTP-Q4_0 head, f16 KV, and the BF16 projector. Measured MTP-vs-none A/B, the client contract, and the rollback state as of 2026-10-03."
tags:
- llm
- frank-dragoon
- dankclaw
- llama-cpp
- rocmfpx
- gfx1151
- gguf
- spec-decode
- mtp
timestamp: 2026-10-02
---
# frank-dragoon — ROCmFPX Q8_0_ROCMFPX_AGENT + MTP (gfx1151, retired)
> **RETIRED 2026-10-03** — superseded by
> [[frank-dragoon-rocmfpx-q6agent-uncensored-mtp]]: Frank swapped to the
> uncensored Q6_0_ROCMFPX_AGENT file (protected heads, one rung down), which
> measures **27.6–28.1 t/s** at a deeper prompt than this file's 17.0. This
> runbook stays live as the rollback reference and the trap catalogue; the
> payload it documents remains on disk at
> `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/`.
>
> Cut over 2026-10-02 from [[frank-dragoon-gufo-qwen38-27b]] (Q4_K_XL, ~15 t/s)
> on two requests from Frank: *highest quality that runs*, and *one full
> context*. It delivered 8.39 bpw weights — two quant rungs above what gufo
> could load — at 17.0 t/s, so quality was not bought with speed.
>
> The capability that forced this whole shape is a GGUF tensor type:
> **`GGML_TYPE_Q8_0_ROCMFPX` (103)**. It exists in exactly one loader on this
> host, so the runtime was decided by the file, not by preference.
## Identity
| | |
|---|---|
| Host / device | Dankclaw (192.168.0.143), Radeon 8060S iGPU `gfx1151`, ~114 GiB GTT aperture |
| Endpoint | `http://192.168.0.143:7770` — alias `frank-dragoon`, `/v1` OpenAI surface |
| Runtime | **`kingjones30/ROCmFPX`** llama.cpp fork @ `36e9acd40e10a87cd3c3ef8ec734668757dc8520`, built by [`scripts/build-rocmfpx.sh`](../../scripts/build-rocmfpx.sh) → `/home/dank/GitPackages/kingjones-ROCmFPX/build/bin/llama-server` |
| Unit | root systemd `frank-dragoon.service` ([`scripts/frank-dragoon.service`](../../scripts/frank-dragoon.service)), `User=dank`, `Restart=always` |
| Serve script | [`scripts/serve-frank-dragoon.sh`](../../scripts/serve-frank-dragoon.sh) (owns the argv), tunables [`scripts/frank-dragoon.env`](../../scripts/frank-dragoon.env) |
| Weights | `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` — 28,214,425,952 B, sha256 `f3ef2bdbf56e244f…` |
| Draft | `…/mtp-Qwen3.8-27B-Q4_0.gguf` — 1,680,271,648 B, sha256 `051a1764cff8c4f3…` |
| Projector | `…/mmproj-Qwen3.8-27B-BF16.gguf` — 931,145,888 B, sha256 `de2a4986…` |
| Fetch | [`scripts/download-frank-dragoon-kingjones.sh`](../../scripts/download-frank-dragoon-kingjones.sh) (`-f`, byte-count + LFS sha256 verified, `--verify-only`, token never on the command line) |
| Seam | `tests/frank_dragoon_unit.bats` (19 tests: payload gates, deployer/serve parity, spec block, KV dtype, sweep identity) |
## Why this file, and why only this runtime
`kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` ships several rungs. Parsed
headers of the two candidates:
| File | ftype | Tensor mix | Output-head tensors at plain Q8_0 | Card MTP acceptance |
|---|---|---|---|---|
| `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` | **115** | 353 F32 + 187 Q8_0 + **311 type-103** | **187** | 0.953 |
| `Qwen3.8-27B-Q8_0_ROCMFPX.gguf` | 111 | 2 Q8_0 + type-103 | 2 | 0.911 |
Two consequences, both measured off the header rather than guessed:
1. **Quality.** The AGENT routing variant keeps the output-head (and other
routing-sensitive) tensors at plain Q8_0 instead of folding them into the
ROCmFPX kernel layout — 187 vs 2 — so the logits the sampler reads are the
highest-precision tensors in the file. It is also the card's MTP-tuned
variant, so it is the *faster* drafter base at the same time.
2. **The runtime is forced.** Type 103 is a private extension: stock llama.cpp
and gufo both fail at load (`unknown tensor type`). Verified on the host
before deploying — the bake-off was not a horse race, it was a lock-in
check.
**The build identity matters as much as the fork.** The 2026-08-25 ROCmFPX
builds read this file and then died in `ggml_compute_forward_set_rows` — the
exact failure documented in [[kingjones-q8agent-rocmfpx]] when this payload was
first attempted on Dankclaw and routed to Dankserver as
[[frank-medium-rocmfpx]] instead. `scripts/build-rocmfpx.sh` therefore checks
the commit out **by SHA** and refuses to build anything else; the pin is
load-bearing, not tidiness.
## Shape and budget
One session, the model's full native window, unquantized KV:
| Dial | Value | Why |
|---|---|---|
| `--ctx-size` | 262,144 | "one full context" — the native window, no rope scaling |
| `--parallel` | 1 | one conversation, all memory to it |
| KV dtype | **f16** (both K and V) | ~64 KiB/token → ~16 GiB; free inside a 114 GiB aperture, so there is no reason to pay a quant |
| `--fit off` | on | the auto-fitter would silently resize the window we asked for |
| `--flash-attn on` | on | required for the long-window attention path to fit |
| `--direct-io` | on | bypasses page cache for the 26 GiB weight stream (mmap is disabled by it — deliberate) |
| `--ctx-checkpoints 16` | 16 | slot state for checkpoint/rollback |
| `--cache-ram 8192` + `--cache-idle-slots` | 8 GiB | prompt cache; verified a repeat pass over the same 6k prompt replays in **0.5 s** |
| `--jinja` | on | without it `chat_template_kwargs` are silently ignored |
| `--reasoning on --reasoning-format deepseek --reasoning-budget -1` | | thinking in its own channel, unbounded |
Measured steady state (production, one model copy resident):
`mem_info_gtt_used` **63.1 GiB** of the ~114 GiB aperture, `mem_info_vram_used`
0.17 / 2.0 GiB (the iGPU's dedicated carve-out is a stub; everything real lives
in GTT). llama-server RSS 2.1 GiB. Boot to `/health` 200: **16 s** — the fastest
endpoint bring-up in the fleet, because there is no engine compile step.
## Speculative decode — measured, not assumed
```
--spec-type draft-mtp --model-draft <mtp-Qwen3.8-27B-Q4_0.gguf>
--spec-draft-n-max 4 --spec-draft-n-min 0 --spec-draft-p-min 0.0
--spec-draft-device ROCm0 --spec-draft-ngl 99
```
MTP in llama.cpp is a property of the file, not of the flags: the head is a
trained tensor set (`nextn_predict_layers = 1` in this repo's head, arch
`qwen35`, 65 blocks) and the draft context runs on the main model's weights, so
there is no second 26 GiB copy in memory. See the fleet law in
`AGENTS.md → llama.cpp serving`.
**A/B, 2026-10-02** — same session, temp 0, 2,202-token prompt, 512 tokens,
`cache_prompt: false`, `frank-dragoon.service` stopped for every scratch arm (the
benchmarking law), baseline arm stripped of `--model-draft` **and** every
`--spec-*` flag:
| Arm | Decode t/s (2 runs) | Prefill t/s | Draft acceptance |
|---|---|---|---|
| **MTP, n-max 4 (deployed)** | **17.00 / 17.14** | 350.6 / 324.9 | 0.462 (mean acceptance length ≈ 1.86) |
| no draft | 7.60 / 7.66 | 381.8 / 381.0 | — |
**+123 % decode for an 8–15 % prefill tax.** Notes that matter before someone
re-litigates this:
- Acceptance here (0.462) is nowhere near the card's 0.953. The card measures
its own short, repetitive text; this is a 2.2k-token prose prompt. Treat the
card's number as a file property, not a throughput promise.
- `n-max 4` is where the gain was measured; 2 and 8 are **unmeasured**. The
gufo-era law applies in spirit — never rank spec settings on a
self-predictable corpus.
- End-to-end on the chat path is slower than the `/completion` number because
thinking tokens are included: a 2,253-token chat prompt generating 512
tokens took 43.8 s wall (11.7 t/s end-to-end) — that is the cost of an
unbounded reasoning budget, not a regression in decoding.
## Client contract (live-verified 2026-10-02)
| Behavior | Observation |
|---|---|
| Model id | `/v1/models` → `frank-dragoon` |
| Tools | native: `get_weather` → `{"city": "Seattle"}` in `tool_calls` |
| Thinking | in **`reasoning_content`** (llama.cpp field — *not* vLLM's `reasoning`); `models.yml` sets `reasoningContentField` accordingly |
| `reasoning_effort` | accepts **only** `xhigh`, `medium`, `low`; anything else is HTTP 500 (the template raises, it does not clamp). `models.yml` folds `high`/`max` → `xhigh` |
| Sampling | server defaults = the card's thinking recipe (temp 1.0, top_p 0.95, top_k 20, min_p 0, presence 0), mirrored in the harness `extraBody` |
| Vision | the BF16 projector loads (`loaded multimodal model`); a red square answers **`Red`** (`usage: 75 prompt / 71 cached`) |
| Prompt cache | second pass over the same 6k prompt: 0.5 s wall |
Three traps, all hit for real:
1. **`content: null` when reasoning eats the budget.** With
`--reasoning-parser`-style separation, a request whose `max_tokens` is
consumed entirely by thinking returns `message.content = None`. A bench
client that indexes `content` crashes *after* measuring and loses the
numbers — fall back to `reasoning_content`, or size the budget (200 was
enough for the vision probe).
2. **`MTP boundary missing for seq_id=… ; resyncing after a non-token step`**
appears in the log under draft resets. Self-recovering; not an outage signal.
3. **Never `pkill -f llama-server` on Dankclaw** — frank-zealot is also a
`llama-server`. The sweep key here is the payload name:
`pkill -f 'Qwen3.8-27B-Q8_0_ROCMFPX_AGENT'`.
## Operations
```bash
# deploy (planner selects this endpoint only via its own unit/env/serve trio)
./scripts/deploy.sh dankclaw --service frank-dragoon
# status / logs / cause of a crash loop
systemctl status frank-dragoon
journalctl -u frank-dragoon -n 200 --no-pager
# rebuild the pinned fork (only when deliberately moving the pin)
scripts/build-rocmfpx.sh
# re-verify the payload without re-downloading
scripts/download-frank-dragoon-kingjones.sh --verify-only
```
`serve-frank-dragoon.sh` refuses to start — before touching the port — unless the
fork binary, the weights, the MTP head and (unless `FRANK_DRAGOON_MMPROJ=""`) the
projector all exist; `scripts/deploy-remote.sh` mirrors the same predicate so a
broken payload never stops the running unit. Caller environment **overrides**
`frank-dragoon.env`, and defaults use `:=` so an explicitly empty value surfaces
as the error it reads as.
## Rollback (as of retirement, 2026-10-03)
**Primary rollback: the ALIGNED Q8 file itself is the previous shape's
payload and remains on disk** at
`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/` — revert the
2026-10-03 swap commit (unit/env/serve trio) or repoint
`FRANK_DRAGOON_MODEL_FILE`/`LLM_DRAGOON_MODEL_FILE`, verify with
`scripts/download-frank-dragoon-kingjones.sh --verify-only`, redeploy.
git tag **`frank-dragoon-gufo-q4kxl-20261001`** (gufo v0.1.1, Q4_K_XL +
DFlash2) is now a **re-download** rollback: the gufo payloads
`/home/dank/models/Qwen3.8-27B-UD-Q4_K_XL/` (16.35 GiB) and
`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL/` (incl. the DFlash2 drafter) were
removed from disk on 2026-10-03 with Frank's explicit approval to free space.
`scripts/build-gufo.sh` remains on disk; re-fetch per
[[frank-dragoon-gufo-qwen38-27b]].
The one-day vLLM MiMo era's payloads (MiMo-V2.6 BF16 18 GiB,
Qwen3.5-9B MTP source 19 GiB, both DFlash draft dirs) were staged to
`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 —
recoverable until the trash is purged, and never part of a rollback plan.
## When it misbehaves
- **`unknown tensor type` / ftype 115 refused** — something moved off the
ROCmFPX binary. Check `readlink -f $(systemctl show frank-dragoon -p ExecStart)`
and the fork commit; this file cannot run on stock llama.cpp or gufo.
- **SIGSEGV in `ggml_compute_forward_set_rows` during load** — a fork build
older than the `36e9acd` pin; rebuild with `scripts/build-rocmfpx.sh`.
History: [[kingjones-q8agent-rocmfpx]].
- **Slow decode with no log error** — check whether the draft is engaged
(`/metrics`, `draft_n` / `draft_n_accepted`; a zero draft count means the head
did not load), and check GTT pressure with
`cat /sys/class/drm/card1/device/mem_info_gtt_used` — a second resident copy
of this model (an orphaned scratch server) drives the whole host into
eviction and halves the rate silently.
- **`500` naming Jinja** — a `chat_template_kwargs` field reached a server
without `--jinja`.
- **HTTP 500 naming the template on the first request of a role** — an
unmapped `reasoning_effort`; fix the fold in `models.yml`, not the model.
serverAppConfigs-docs/LLM/frank-dragoon-unsloth-qwen38-27b-q8xl.md
-117
View File
@@ -1,117 +0,0 @@
---
type: reference
title: "stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-dragoon)"
description: "The RETIRED frank-dragoon runbook (2026-09-21 → 2026-09-28; replaced by [[frank-dragoon-gufo-qwen38-27b]], and the rollback target) — the stock llama.cpp prebuilt (unsloth b11030, ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.46 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) and q8_0 KV on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the full-window/one-stream shape, the DFlash2 spec method, the Qwen-Sharp template, the payload, the client contract, and the failure modes worth knowing."
tags:
- llm
- llama.cpp
- unsloth
- dankclaw
- frank-dragoon
- strix-halo
- gfx1151
- q8
- dflash2
- rocm
- q8-kv
timestamp: 2026-09-21
---
# frank-dragoon — stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2)
> **RETIRED 2026-09-28**, replaced by [[frank-dragoon-gufo-qwen38-27b]] (the
> gufo engine, same weights, same port and alias); the prebuilt is intact on
> disk as the rollback target. ACTIVE from 2026-09-21, when it replaced
> [[frank-dragoon-kairic-edge-qwen38-27b]] (Kairic Edge IU4 W4A4) and
> [[frank-dragoon-unsloth-qwen38-flash-next]] (the Flash-Next MoE it
> preceded).
## Identity
- **Endpoint:** `frank-dragoon` at `192.168.0.143:7770` (Dankclaw).
- **Runtime:** stock llama.cpp — the unsloth `b11030` prebuilt
(`/home/dank/GitPackages/unsloth-llama-b11030/llama-server`), ROCm gfx1151.
A mainline build, not a fork: it supports the `qwen3_5` architecture and the
DFlash2 drafter (`llama_model_qwen35`, `llama_model_dflash`,
`common_speculative_impl_draft_dflash`).
- **Service shape:** root systemd unit `scripts/frank-dragoon.service`
(`User=dank`), launched by `scripts/serve-frank-dragoon.sh` with tunables from
`scripts/frank-dragoon.env`.
## Shape
- **Model:** `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` — a 27B **dense** model,
31.46 GiB GGUF (sha256-verified by `scripts/download-frank-dragoon.sh`).
- **Context:** one stream, full 262,144 tokens (the model's native window),
`PARALLEL_SLOTS=1`.
- **Placement:** all layers on the iGPU (`--n-gpu-layers 999`). The 27B dense
weights (31.46 GiB) + the DFlash2 drafter (1.14 GiB) + q8_0 KV (~8.6 GiB) +
compute buffers fit the iGPU's GTT aperture (~114 GiB).
## Spec method
DFlash2 (a block-diffusion drafter) at **depth 4**. The drafter is the z-lab
`Qwen3.8-27B-DFlash2-Q4_K_M` (1.14 GiB), loaded via `--model-draft` with
`--spec-type draft-dflash` and `--spec-draft-device ROCm0`. It has a sliding
window of 2048, so its draft KV is small (~40 MiB). Greedy verification
(`--temp 0 --top-p 1.0 --top-k 0 --min-p 0.0`) is required.
## KV
**q8_0** (`--cache-type-k q8_0 --cache-type-v q8_0`), ~8.6 GiB at 262k. The
unsloth b11030 build does **not** support fp8 KV (allowed: f32/f16/bf16/q8_0/
q4_0/q4_1/iq4_nl/q5_0/q5_1); q8_0 is the highest-precision 8-bit option.
## Template
The Qwen-Sharp v22.4.0 template is pinned via `--chat-template-file`
(`dankserver/frank-medium/qwen-sharp-v22.4.0.jinja`) with `--jinja` and
`--chat-template-kwargs '{"preserve_thinking":true}'`. Without
`--chat-template-file` the binary falls back to the built-in `peg-native`
template, which produces a ~73,600-token prompt for a ~250-token request.
`preserve_thinking` keeps prior thinking blocks across turns for multi-turn
agents. The template gates `reasoning_effort` to `{xhigh, medium, low}` — the
harness folds its levels onto that set in `models.yml` (`reasoningEffortMap`).
## Payload
`scripts/download-frank-dragoon.sh` fetches the main GGUF and the DFlash2
drafter, each verified against its declared byte count + sha256
(`--verify-only` re-checks). The serve script refuses to start if the binary,
the main GGUF, or the drafter is missing.
## Client contract
- **Alias:** `frank-dragoon` (the `id` at `/v1/models`).
- **Reasoning:** extracted to `reasoning_content` (`--reasoning auto`,
`--reasoning-format deepseek`, unlimited budget).
- **Tools:** supported (`--jinja` is pinned).
- **Vision:** none — text-only (no mmproj), matching the prior frank-dragoon.
- **Sampling:** greedy by default; a client that sends a temperature gets its
own, and the harness applies per-model overrides from `models.yml`.
## Failure modes worth knowing
- **Model load is slow (minutes cold).** Health returns 503 while loading; do
not kill the process. `deploy.sh` waits up to 20 minutes (`LLM_WAIT`).
- **`--ngl` (double-dash) is invalid in b11030** — use `--n-gpu-layers`
(the serve script uses the long form).
- **The 8-bit UD-Q8_K_XL decodes ~17 t/s** vs the prior 4-bit Kairic Edge IU4
at ~22 t/s — the ~23 % larger weight size (31.46 vs 26 GiB) explains the
gap; it is not a regression.
- **DFlash2 draft acceptance is text-variety-dependent** (0.85 on a trivial
one-word answer, 0.519 on a varied explanatory prompt) — do not read a
single-request acceptance as the steady state.
## Rollback
Payload and runtime for [[frank-dragoon-kairic-edge-qwen38-27b]] (Kairic Edge
IU4 W4A4) remain intact on disk. See [[frank-dragoon-unsloth-qwen38-flash-next]]
for the Flash-Next predecessor it replaced on 2026-09-21.
# Citations
- unsloth prebuilts: https://huggingface.co/unsloth
- Model: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
- DFlash2 drafter: https://huggingface.co/z-lab
- Serve script: `scripts/serve-frank-dragoon.sh`; tunables: `scripts/frank-dragoon.env`
serverAppConfigs-docs/LLM/frank-dragoon-unsloth-qwen38-flash-next.md
-47
View File
@@ -1,47 +0,0 @@
---
type: reference
title: "unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on Dankclaw (frank-dragoon)"
description: "The RETIRED frank-dragoon runbook (2026-09-21, same-day swap; replaced by [[frank-dragoon-unsloth-qwen38-27b-q8xl]]) — the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the 200k/one-stream shape, the explicit -md MTP wiring, the payload, the client contract, and the failure modes worth knowing."
tags:
- llm
- unsloth
- dankclaw
- frank-dragoon
- strix-halo
- gfx1151
- qwen4exp
- rocm
- mtp
timestamp: 2026-09-21
---
# frank-dragoon — unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS (RETIRED)
> **RETIRED 2026-09-21 (same-day swap).** This was the ACTIVE frank-dragoon
> runbook on 2026-09-21 before it was replaced by
> [[frank-dragoon-unsloth-qwen38-27b-q8xl]] (27B dense UD-Q8_K_XL) later the
> same day. The payload and runtime remain on disk as the rollback target.
## What it served
- **Runtime:** unsloth llama.cpp prebuilt `b11030-mix-5ff778e` (ROCm
gfx1151), root systemd unit, port 7770.
- **Model:** `unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS` (a 125B MoE / 6B
active), 87.3 GiB shipped as three split parts. The split parts ARE the
payload — concatenating them trips `invalid split file name` (part 1's
header advertises `file_count=3`).
- **Shape:** one 200,000-token conversation (single stream), MTP depth 2 via
the explicit `-md` shared-Q8_0 head. Draft acceptance verified live
0.83–0.92.
- **Memory:** the 114 GiB GTT aperture on the Strix Halo iGPU.
## Rollback
Payload and runtime intact on disk. See [[frank-dragoon-unsloth-qwen38-27b-q8xl]]
for the current ACTIVE runbook and [[frank-dragoon-ciru-orca]] for the
predecessor.
# Citations
- unsloth prebuilts: https://huggingface.co/unsloth
- Model: https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF
serverAppConfigs-docs/LLM/frank-dragoon-vllm-mimo-v26-bf16.md
-186
View File
@@ -1,186 +0,0 @@
---
type: reference
title: "frank-dragoon on Dankclaw — vLLM TheRock, MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (RETIRED)"
description: "The RETIRED frank-dragoon runbook (2026-10-01, one day) — the digest-pinned kyuz0 TheRock gfx1151 vLLM image serving XiaomiMiMo MiMo-V2.6-Distill-Qwen-9B in full BF16 on the Strix Halo iGPU. Kept for the measured record: the roof math, the cross-model MTP drafter, and the autoresearch finding that spec decode is NET-NEGATIVE on real chat+reasoning traffic here (6.19 t/s with MTP-2 vs 9.01 t/s without). Rolled back to [[frank-dragoon-gufo-qwen38-27b]] by Frank the same night."
tags:
- llm
- frank-dragoon
- dankclaw
- vllm
- strix-halo
- mimo
- mtp
- spec-decode
- bf16
timestamp: 2026-10-01
---
# frank-dragoon — vLLM MiMo-V2.6-Distill-Qwen-9B BF16 (cross-model MTP)
> **RETIRED 2026-10-01 night, one day active.** Frank's call after the
> autoresearch loop: battery decode 6.19 t/s (MTP-2) / 9.01 t/s (no draft)
> against gufo's ~15 t/s on the 27B — and every ≥6-bit quant path in vLLM
> 0.30 is dead on gfx1151 (quark int8 rejected; all FP8 kernels CDNA-only;
> torchao config unwired). Back to [[frank-dragoon-gufo-qwen38-27b]]
> (Q4_K_XL, 1 x 262144). The measured content below stands as the record.
## Identity
- **Endpoint:** `frank-dragoon` at `192.168.0.143:7770` (Dankclaw).
- **Runtime:** vLLM 0.30.0 in the **digest-pinned** `docker.io/kyuz0/vllm-therock-gfx1151`
image (image ID `2d24a8c5a3fd`, digest `sha256:98c1f3ac9833…` — ROCm 10.0.0 /
torch 2.11.0 / TheRock, Triton Gated DeltaNet kernels). The official vLLM
images are CUDA-only; this fork is the fleet's gfx1151 path. Plain
`podman run` driven by `scripts/serve-frank-dragoon.sh` under a root unit
(`scripts/frank-dragoon.service`, `User=dank`, `Restart=always`,
`ExecStop=podman stop -t 120`, `TimeoutStopSec=180` > the stop grace).
- **Model:** `XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` in **full BF16**
(17.53 GiB, 4 shards) — the same checkpoint frank-zealot serves as an INT8
W8A16 AutoRound conversion on the 3090, here unquantized.
- **Draft:** `Qwen/Qwen3.5-9B` (18 GiB on disk; only the 15 `mtp.*` tensors
are loaded into the draft pass).
## Why this shape (the roof math)
Decode speed for a dense model on Strix Halo is bandwidth-bound:
BF16 weights x one pass per token against the LPDDR5x aperture. Measured
2026-10-01 on this host: **13.1 t/s is ~94% of the roof** — the payload, not
a misconfiguration. The same INT8 payload reaches 83.9 t/s on the 3090
because it moves ~12 GiB through ~936 GB/s. The only way past a bandwidth
wall is generating more than one token per weight pass: **speculative decode
is the performance feature of this endpoint**, not a garnish.
The memory side is the opposite of a squeeze: KV grows ONLY on the 8 full
attention layers (32 KiB/token bf16: 8 x 2 x 4 heads x 256 dim x 2 B);
the 24 Gated DeltaNet layers hold a FIXED ~25 MiB state per running
sequence. `--max-model-len 262144` (the native window) with
`--max-num-seqs 4` at `--gpu-memory-utilization 0.45` boots clean — the
four-session contract the gufo shape ran, with the pool shared by the
scheduler instead of carved per session.
## Spec method: cross-model MTP — measured, not assumed
| arm | prefill | decode | acceptance |
|---|---|---|---|
| BF16 base | 17,876 t/s | 13.1 t/s | — |
| BF16 + DFlash 8 tok | 7,976 t/s | 17.4 t/s | pos0 59% pos1 26% (token rate 14.8%) |
| **BF16 + MTP 2 tok** | 7,380 t/s | **19.8 t/s** | pos0 71% pos1 44% (token rate 57.4%) |
One session, temperature 0, 28,647-token prefill / 512-token decode, same
client for all three arms (2026-10-01 bench window, frank-dragoon stopped per
Benchmarking Policy). **MTP wins (+51% vs +33%) and needs only 2 draft
tokens** — acceptance past position 1 is 44% and falling; DFlash's own curve
decays 59/26/14/7, so 8 proposals are mostly wasted verification.
`num_speculative_tokens` beyond 2 is UNMEASURED; do not raise it without a
fresh A/B. Both spec arms pay ~55–59% of prefill (the draft passes share
the compute queues) — this endpoint's thinking-heavy traffic spends its wall
clock in decode, which is the regime where that trade wins.
**Why a Qwen head accepts on a MiMo target:** MiMo-V2.6 is a distillation of
Qwen3.5-9B that preserves the hidden-state geometry — identical
4096-hidden/248320-vocab/32-layer/interval-4 structure, and the 71%/44%
per-position acceptance is the direct measurement that the distillation
preserved more than geometry. MiMo itself ships **ZERO `mtp.*` tensors**
(verified in both the BF16 base and the INT8 conversion) — which is exactly
why frank-zealot's SGLang NEXTN draft is a 0.00-acceptance ghost: SGLang
reads the head from the TARGET checkpoint, vLLM reads the draft from a
separate path. That difference is the whole trick:
```
--speculative-config '{"method":"mtp","model":"/models/Qwen3.5-9B","num_speculative_tokens":2}'
```
The alternative drafter (`z-lab/Qwen3.5-9B-DFlash`, 6 layers, target_layer_ids
[1,5,…,29]) stays fetched on disk; flip `FRANK_DRAGOON_SPEC=dflash` and repoint
`FRANK_DRAGOON_SPEC_DIR` to use it. Its card's "vLLM PR #40898 unmerged" claim
is FALSE — `vllm/v1/spec_decode/dflash.py` ships in 0.30.
## The numerics trap (greedy text is NOT preserved by spec decode)
Measured: temperature-0 completions DIFFER across arms (sha `1318aeb6` base,
`0208b45d` dflash, `20c41c38` mtp). Batched verification changes the bf16
reduction order, argmax near-ties diverge, and the divergence cascades.
vLLM spec decode is distribution-preserving, not bitwise-greedy-preserving.
The gufo-era "greedy == autoregressive text" law does not carry over; do not
chase text drift between spec and no-spec arms as a regression.
## Payload
- Target: 4 safetensors shards (17.53 GiB) + config/tokenizer under
`/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-BF16/`, each shard
byte-count + LFS sha256-verified by
`old/download-frank-dragoon-mimo.sh` (`--verify-only` re-checks; no token on
the curl command line).
- MTP source: `old/download-frank-dragoon-qwen35.sh` →
`/home/dank/models/Qwen3.5-9B/` (4 shards behind
`model.safetensors.index.json`; the shards carry the odd HF asset names
`model.safetensors-0000N-of-00004.safetensors` — the index resolves them,
the engine boots from exactly this tree).
- DFlash drafter: `old/download-frank-dragoon-dflash.sh` →
`/home/dank/models/Qwen3.5-9B-DFlash/`.
- **Payload status 2026-10-02:** all three trees moved to
`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` (59 GiB staged with
the other retired frank payloads) and the fetch trio moved to `old/` — the
endpoint is two runtimes past this one, and re-fetch is cheaper than disk.
- The serve script gates the image by REPO DIGEST (rootless podman resolves
tags from the network; the pin exists to boot exactly these bytes
offline), then config + index + exact shard count, then the spec draft
(either weight layout: single `model.safetensors` or a sharded index),
then `/dev/kfd` + whole `/dev/dri` (never individual renderD nodes —
Dankserver 2026-09-29 lesson).
## Client contract
- **Alias:** `frank-dragoon` (the `id` at `/v1/models`).
- **Reasoning:** `--reasoning-parser qwen3`; vLLM emits the thinking text in
**`reasoning`**, NOT llama.cpp/gufo's `reasoning_content` — the harness
`models.yml` compat block for this provider sets
`reasoningContentField: "reasoning"`. Miss that and thinking silently
vanishes from transcripts.
- **Tools:** `--enable-auto-tool-choice --tool-call-parser qwen3_coder`.
- **Vision:** the checkpoint carries the `qwen3_5_vision` tower but the
serve script passes `--language-model-only` — the endpoint stays
text-only by decision; dropping the flag brings vision back.
- **Sampling:** request-side; the harness sends the Qwen3.5-family recipe
(temp 0.7, top_p 0.8) via `compat.extraBody`, and
`chat_template_kwargs.{enable_thinking,preserve_thinking}` — the
`chat_template_kwargs` form is what vLLM reads; a top-level
`preserve_thinking` is ignored there.
- **Env hygiene:** every tunable is `FRANK_DRAGOON_`-prefixed because HIP
reads bare `TEMP`/`HOST`/`GPU_MAX_HEAP_SIZE` and the serve script exports
the env file with `set -a`.
## Ops
- Boot ~200–240 s warm; health 503 while loading (do not kill it).
- The serve script sweeps stale container identities before starting:
`frank-dragoon-mimo` itself and the bench-window container `vllm-mimo`.
- Container death → `podman wait` returns non-zero from ExecStart →
`Restart=always` self-heals (a clean exit-0 drain also restarts, the
frank-archon lesson).
- Bench windows: `systemctl stop frank-dragoon` is enough (mask fails on the
relinked unit; the deploy restores).
- First-deploy facts (2026-10-01, deploy #3): healthy after **230 s**;
boot log `GPU KV cache size: 819,332 tokens, Maximum concurrency for
262,144 tokens per request: 3.13x` — four sessions share the pool
(the scheduler, not a carve). Live battery green: alias `frank-dragoon`,
reasoning split into `reasoning`, tool_calls with correct args, and
`/metrics` acceptance pos0 69% pos1 44% (54/86 draft tokens accepted),
matching the bench window.
## Rollback
Git tag **`frank-dragoon-gufo-q4kxl-20261001`** — gufo v0.1.1 (built from
source, `scripts/build-gufo.sh` stays on disk) + the Q4_K_XL Qwen3.8-27B
payload + DFlash2 drafter, all still on the host. Restoring the tag's
unit/env/serve trio and redeploying returns the 27B endpoint.
# Citations
- Model: https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B
- Draft source: https://huggingface.co/Qwen/Qwen3.5-9B
- DFlash drafter: https://huggingface.co/z-lab/Qwen3.5-9B-DFlash
- Engine image: https://hub.docker.com/r/kyuz0/vllm-therock-gfx1151
- Serve script: `scripts/serve-frank-dragoon.sh`; tunables: `scripts/frank-dragoon.env`
- Predecessor: [[frank-dragoon-gufo-qwen38-27b]]; sibling on the same arch:
[[frank-zealot-sglang-mimo-v26]]
serverAppConfigs-docs/LLM/frank-dragoon.md
+236
View File
@@ -0,0 +1,236 @@
---
type: reference
title: "frank-dragoon on Dankclaw — halogen-flash-server 0.16.2, native v2 abliterated Qwen3.8-Flash-Next (canonical)"
description: "The one runbook for frank-dragoon, the Strix Halo iGPU endpoint (192.168.0.143:7770): current shape from the env, every retired era with its rollback path, the durable laws this endpoint paid for, the wire contract, and the files that own it."
tags:
- llm
- frank-dragoon
- dankclaw
- strix-halo
- gfx1151
- halogen
timestamp: 2026-10-18
---
# frank-dragoon — halogen-flash-server 0.16.2, native v2 abliterated (Dankclaw iGPU)
Strix Halo (gfx1151) endpoint `http://192.168.0.143:7770`, served id `frank-dragoon`: a podman container
under a root `User=dank` unit. Live shape since 2026-10-17, pool raised to 401,408 on 2026-10-18
(`2b12716`). This page replaces every per-era dragoon runbook; the Eras table is the rollback record.
## Current shape
`scripts/frank-dragoon.env` is authoritative and `scripts/serve-frank-dragoon.sh` passes exactly those
flags and nothing else. Payload `/home/dank/models/halogen-v2-abliterated/`, verified by
`scripts/download-frank-dragoon-v2abl.sh --verify-only`:
| File | Bytes | sha256 | Source |
|---|---|---|---|
| `qwen38-flash-next-v2-abliterated.hgn` | 66,687,678,432 | `b5bd925f…` | Quat3rnion `@main` — the NATIVE 2026-10-11 v2 conversion of orcarouter's BF16 Qwen3.8-Flash-Next, abliterated |
| `qwen38-flash-next-ngram.hgn` | 51,200,246,144 | `3450acada…` | peonist `@5cc17cea`, **REQUIRED**, hardlinked |
| `qwen38-flash-next-vision.hgn` | 897,916,416 | `d62e0ae5…` | peonist `@e053f488`, hardlinked — vision ON |
| `tokenizer/` (6 flat files) | ~22 MB | `tokenizer.json` LFS | peonist `@e053f488`, hardlinked |
Image `ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a`
(0.16.2, closed source), container `frank-dragoon-halogen`, published `7770:8731`; the engine's
unauthenticated token port 8730 stays on container loopback. Engine stdout →
`/home/dank/logs/frank_dragoon.log`, **not** the journal.
- Shape: `HALOGEN_CTX=200704` hard per-request ceiling · `HALOGEN_KV_SLOTS=2` ·
`HALOGEN_KV_POOL_POSITIONS=401408` = slots × ctx, the top of the band `tests/frank_dragoon_unit.bats`
enforces (`ctx ≤ pool ≤ slots × ctx`) · KV ~28 KiB/position → **11.0 GiB reserved** · no KV-dtype knob
exists here, the pool is the only KV lever · native window 262,144, `HALOGEN_ROPE_YARN` beyond it.
- Knobs: `HALOGEN_MAX_TOK=8192` sizes the prefill arena (5.7 GiB) and is only the default for requests
omitting `max_tokens` · `HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` are a coupled
pair · MTP depth 2 with the head INSIDE the checkpoint (**`HALOGEN_MTP_HEAD` stays unset**) and
`HALOGEN_SPEC_ADAPT=0` · sampler 1.0 / 0.95 / 20 server-side · `HALOGEN_WEIGHTS_LOCK=1` ·
`HALOGEN_HOST_RESERVE_GIB=20` · prompt cache ON by image default (`PROMPT_CACHE=2`, no flag
passed): 12 entries, which the boot ledger prices at 111 MiB each at this window = 1.3 GiB
(the env header's "~85 MiB here" estimate did not survive that ledger line).
- Deployed boot ledger: `62.9 GiB of weights locked in RAM (0.8 of them the vision tower's), 11.0 GiB
of KV pool, 5.7 GiB of working memory, 79.6 GiB in all`; n-gram `mapped 51.2 GB, NOT pinned
(Pin::None)`; no clamp line. Host `MemAvailable` read 22 GiB right after the 2026-10-18 raise —
~20 GiB above the 20 GiB reserve, against 10–15 GiB of baseline co-tenancy (frank-zealot, ComfyUI,
Caddy/observability).
- Harness (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw)`): `contextWindow: 155648`
(= ceiling − 45k per the 400-zone law, never higher), `maxTokens: 45056`, `input: [text, image]`,
`reasoningContentField: "reasoning_content"`, `high→xhigh` fold, `extraBody` 1.0/0.95. Compaction
rides [[frank-zealot]].
- Deploy `./scripts/deploy.sh dankclaw --service frank-dragoon`; gate = boot-ledger greps (weights,
pool, MTP, prompt cache, **no clamp**) + `/health` 200 + a tool call + the 512 px red-PNG probe.
Kernel: `CONFIG_HSA_AMD_SVM=y` (CachyOS ships it) and `amdgpu.noretry=0` — the env says check it at
deploy; boot line per [[dankclaw-iommu-translation-off]].
## Eras
Rollback = revert that era's swap commit (or repoint the payload vars) and redeploy
`dankclaw --service frank-dragoon`. Newest first; payloads staged under `~/.AGENTTRASH/` are **not** a
rollback plan — recoverable only until the trash is purged.
| Window | Payload | Runtime | Shape | Replaced because | Rollback |
|---|---|---|---|---|---|
| 2026-10-06 → 10-17 | Swift1.5 abliterated v2 `.hgn` r2 66,687,678,464 B `9c880b8b…` + same trio | flash-server 0.16.2, same digest | 2 slots, ctx 262,144, pool 524,288 (14.6 GiB), MTP always-on | 2026-10-17 A/B at that shape: NIAH 18/18 vs 17/18, refusal census 0/5 on both, tool calls + red-PNG probe identical, needle hit at both depths of a 21.5k haystack, decode 50.0 vs 46.8 t/s, cold prefill 1,604 vs 1,425 t/s, PPL 3.968 vs 3.877 (finetune's edge gone: +2.3 %), thinking 1,476 vs 784 mean, 7.4 GiB RAM back | `/home/dank/models/halogen-flashnext-v2/` + `scripts/download-frank-dragoon-flashnext.sh` |
| 2026-10-05 → 10-06 | peonist `halogen-qwen3.8-27b` `qwen3.8-27b-p1w4d-d2.hgn` 35,865,565,184 B `274c3fc7…` | halogen 0.1.4 `sha256:dc0a39a0…` | 1 slot × 200,700, DFlash2 drafter, `HALOGEN_CACHE_MB=16384`, text-only, censored base | flash-server v2 restored vision, abliteration and MTP | `/home/dank/models/halogen-qwen3.8-27b/` + `scripts/download-frank-dragoon-halogen.sh`; 0.1.4 digest still in podman storage |
| 2026-10-03 → 10-05 | kingjones777 Uncensored `Q6_0_ROCMFPX_AGENT.gguf` 24,856,607,008 B `e966deb3…` + `mtp-Q4_0` `051a1764…` + BF16 mmproj | `kingjones30/ROCmFPX` @ `36e9acd` (`scripts/build-rocmfpx.sh`) | 1 × 262,144 f16 KV, `draft-mtp` n-max 4, vision on | halogen card claimed 31.7 t/s vs 27.6–28.1 measured | `/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/` + `/home/dank/GitPackages/kingjones-ROCmFPX/build` + `download-frank-dragoon-kingjones-uncensored.sh` |
| 2026-10-02 → 10-03 | aligned `Q8_0_ROCMFPX_AGENT.gguf` 28,214,425,952 B `f3ef2bdb…` | same fork pin | 1 × 262,144 f16 KV, `draft-mtp` 4 (MTP +123 % decode) | the uncensored file was faster *and* uncensored | `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/` + `download-frank-dragoon-kingjones.sh` |
| 2026-10-01 (one day) | MiMo-V2.6-Distill-Qwen-9B BF16 17.53 GiB + cross-model MTP from `Qwen3.5-9B` | vLLM 0.30 TheRock `kyuz0/vllm-therock-gfx1151` `sha256:98c1f3ac…` | 4 seqs × 262,144 on a scheduler-shared pool, 2 draft tokens | 6.19 t/s with MTP / 9.01 without vs gufo ~15, and no ≥6-bit vLLM quant path boots on gfx1151 | staged `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` until purged; fetch trio in `old/` |
| 2026-09-28 → 10-01, again 10-01 → 10-02 | Swift-1.5 Uncensored `UD-Q8_K_XL` 31,457,990,272 B `9076f99a…`, then `UD-Q4_K_XL`; z-lab DFlash2 Q4_K_M | gufo v0.1.1 (`b0f8673`), source build | 1 × 262,144, `--draft-policy adaptive --draft-tokens 7`, no KV knob, no Jinja | ROCmFPX gave 8.39 bpw + MTP at 17 t/s | payloads **deleted 2026-10-03** → re-download `download-frank-dragoon-swift.sh` (+ `download-frank-dragoon.sh` for the drafter); tag `frank-dragoon-gufo-q4kxl-20261001`; `scripts/build-gufo.sh` |
| 2026-09-21 → 09-28 | unsloth `Qwen3.8-27B-GGUF:UD-Q8_K_XL` 31.46 GiB + DFlash2 Q4_K_M 1.14 GiB | unsloth llama.cpp `b11030` prebuilt | 1 × 262,144, q8_0 KV, Qwen-Sharp v22.4.0 template | gufo: +37 % prefill shallow, +80 % at 131k, 4.5× faster load | `/home/dank/GitPackages/unsloth-llama-b11030/` on disk; payload deleted 2026-10-03 → `download-frank-dragoon-unsloth.sh` |
| 2026-09-21 (same day) | unsloth `Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS` 87.3 GiB in 3 split parts | same prebuilt | 1 × 200,000, MTP depth 2 via explicit `-md` shared Q8_0 head | swapped to the 27B dense hours later | payload + prebuilt on disk at retirement; the parts ARE the payload (concatenating → `invalid split file name`) |
| 2026-09-21 (same day) | `jcbtc/Qwen3.8-27B-IU4-Kairic-Edge` + 3 `.pfs` = 26.05 GiB | ciru-ai/ROCmFPX `kairic-edge-qwen38-27b-v1.2` @ `205a3e5` | 1 × 262,144, native MTP 4 + n-gram | same-day swap to the 27B dense | payload + runtime intact on disk at retirement |
| 2026-09-19 → 09-21 | `jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca` | CIRU v4.4.1 (QSA window hotfix) | never written down — that runbook is a frontmatter stub | the stock llama.cpp prebuilt won | `scripts/download-frank-dragoon-orca.sh` |
| 2026-09-11 → 09-15, again 09-16 → 09-19 | `peonist-ai/halogen-qwen3.8-flash-next` w4b 115.55 GiB + in-place-versioned `overlay.hgn` | halogen-flash-server 0.5.8, rootless Quadlet | 2 × 200,192 over a 400,384 pool, `MAX_TOK 24576`, `HOST_RESERVE 20`, vision OFF | displaced by CIRU Orca; the shape starved co-tenants | re-tag 0.5.8 + first-start `HALOGEN_DOWNLOAD` re-fetch into `/home/dank/models/halogen`; the Quadlet left the repo 2026-10-05, so the unit must be rewritten |
| 2026-09-15 → 09-16 (one day) | `mradermacher/Qwen3.8-Flash-Next-Uncensored` i1-Q4_K_M 111 GiB + f16 mmproj | ik_llama.cpp `d85a1d7`, 3090 + CPU hybrid | 1 × 262,144, `-ot exps=CPU`, `GGML_CUDA_NO_PINNED_WEIGHTS=1` | 3.8–4.0 t/s: ~100 CPU↔GPU graph splits per token, GPU at 5 % SM | `/home/dank/models/mradermacher-Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`, `/home/dank/GitPackages/ik_llama.cpp/build`, trio `scripts/.AGENTTRASH/frank-dragoon.{env,service}.ik-llama.cpp` + `serve-frank-dragoon.sh.ik-llama.cpp` |
| 2026-08-30 → 09-11 | kingjones STRIX `Q4_0_ROCMFP4-STRIX.gguf` 113.47 GiB (51.2 B PLE via mmap) | ROCmFPX @ `36e9acd` | 2 × 200,704 over one 401,408 f16 window, spec off, f16 mmproj | halogen-flash-server ≈ 4× prefill | payload dir + `/home/dank/GitPackages/kingjones-ROCmFPX/build`; trio `.AGENTTRASH/frank-dragoon.{env,service}.llama.cpp` + `serve-frank-dragoon.sh.llama.cpp` |
| 2026-08-25 → 08-29 (restored 08-29 → 08-30 after the Q8_0 AGENT file crashed before it ever served) | `kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP`: `Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf` 23.15 GiB (private ftype 114) + a **SEPARATE** MTP head `mtp-Qwen3.8-27B-Q4_0.gguf` 1.56 GiB via `--model-draft` + `mmproj-Qwen3.8-27B-Q8_0.gguf` 0.59 GiB | CIRU ROCmFPX llama.cpp (`build-promptforge`) | 1 slot at `-c 196608`, `-b 8192 -ub 8192 -ctxcp 32`, `-fa on --no-mmap`, f16 K+V, `draft-mtp` n-max 4 with the head on ROCm1 at f16 KV, QwenSharp template (`enable_thinking`, `preserve_thinking`) behind the `peg-native` + `deepseek` chat/reasoning pairing the Pi agent expected | the Q8_0 AGENT experiment, then the STRIX Q4_0 tier | `/home/dank/models/kingjones-Qwen3.8-27B-STRIX-MTP/` (six card files + projector), runtime `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` |
| 2026-08-29 → 08-30 (one day) | agentionai FAST-imatrix **`v2/` joined-PLE** file 87.06 GiB | charlie12345/ROCmFPX `build-qwen38` + `scripts/patches/qwen4exp-qsa-f32-mask.patch` | 2 × 200,704 / 401,408 f16 KV, no MTP head loads | kingjones STRIX on its own pinned fork | `/home/dank/GitPackages/ROCmFPX/build-qwen38` + the agentionai payload, both on disk |
| 2026-08-18 → 08-25 | caf-eemin `Q4_0_ROCMI4.gguf` 13.50 GiB `ad51ecbe…` with embedded NextN/MTP | charlie12345/ROCmFPX, ROCmI4 W4A4 | one window, MTP `n4 / p0.60` | the 27B STRIX tier took the slot | re-fetch into `/home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP/` |
| through 2026-08-24 | CIRU ActiveFPX PromptForge v3: GGUF 15.44 GiB + 3 `.pfs` + mmproj ≈ 36.7 GiB | CIRU ROCmFPX `build-promptforge` | 1 × 262,144, native MTP depth 4, ngram off | eemin ROCmI4 | `/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/` + `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`; scripts `git revert 4c43839 7aee1df 40d50fe` or checkout `132e147` |
| 2026-08-22 (same day) | `jcbtc/Qwen3.8-27B-IU4-Kairic-Edge` 15.48 GiB + 3 `.pfs` | ciru-ai/ROCmFPX tag `kairic-edge-qwen38-27b-v1` @ `e97b324` | one window, IU4 W4A4 lane, ubatch 512 | prefill 1.5–2.3× faster but prose decode 15–26 % slower → back to v3 the same day | `/home/dank/models/Kairic-Qwen3.8-27B-IU4-Edge/` + `/home/dank/ciru-qwen38/ROCmFPX-kairic/build-kairic/bin/llama-server` |
| 2026-08-21 (same day) | unsloth `UD-Q8_K_XL` 31.4 GiB + z-lab DFlash2 Q4_K_M 2.0 GiB | upstream llama.cpp PR #27342, `GGML_VULKAN=ON` | Vulkan0 — HIP has no `TOP_K`, so DFlash could not run there | slower than the CIRU runtime it replaced | both files in `/home/dank/models/`; the PR build must be rebuilt |
Port note: this slot settled on **:7770** on 2026-08-22 (52625 before it); the 2026-08-17 fleet redo
created it — earlier topologies in [[frank-redo-2026-08-17]].
## Laws learned here
- **KV arithmetic:** ~28 KiB/position incl. block scratch plus ~115 MiB/slot of O(1) state; reservations
measured 5.5 GiB @200,704, 7.2 @262,144, 11.0 @401,408, 14.6 @524,288. Quote the `reserving the KV
pool` line, not the preflight formula.
- **One pool, not a partition:** the pool decides how many conversations stay resident; only
`HALOGEN_CTX` sets the window, and over-window is a 400 naming the limit, never a truncation. The
2026-10-18 doubling to 401,408 was bought as pool parity with [[frank-zealot]] so each slot can
hold a maximum-length request at once — it did not lengthen the window by one token.
- **Budget:** device budget = 121.2 GiB (unrounded `MemTotal` 121.2159) − pinned weights −
`HOST_RESERVE_GIB`; need = KV + arena + vision, arena = 3.80 + 8.40 × (`MAX_TOK`/16384) GiB — it
predicts every recorded point to 0.1 GiB. `HOST_RESERVE_GIB` is a term in that inequality, **not an
allocation**: it cannot return RAM, only take the pool away (ceiling 16 at the 524,288 shape, ≥18
clamped it). Free RAM by cutting the pool or `MAX_TOK`.
- **`HALOGEN_KV_POOL_FIT` prints its sizing block only when it clamped** — silence is not headroom and a
green `/health` is not the pool you asked for.
- **RAM the pool takes is RAM the 47.68 GiB n-gram table loses** (page cache, `Pin::None`). The real
failure is file-cache starvation, not RAM pressure — long prompts crawl with the disk busy and the
process uninterruptible, and the fix is a *smaller* pool.
- Pinned weights count as reclaimable file cache, so `free`, `MemAvailable` and Grafana overstate free
RAM by ~63 GiB; believe the engine's `… in all` line.
- **Pin gate:** the engine refuses to pin below checkpoint + 16 GiB of `MemAvailable` (pinned pages
cannot swap, and under the floor the n-gram table pages in from disk every long prompt), so a
RAM-short host crash-loops the unit every ~5 s per boot — 2026-10-06: 8 boots at `MemAvailable`
74.77 GiB against the 78.11 GiB needed, self-resolved in ~2 min when the dying 27B engine freed RAM.
The serve script waits `FRANK_DRAGOON_RAM_WAIT` (600 s) and refuses loudly if the RAM never clears.
- `HALOGEN_FLASH_PIN_TRUNK=0` is refused by the v2 loader (and cost several times decode on the w4b
build): disk-backing the weights is not a lever here.
- **MTP is always on:** `HALOGEN_SPEC_ADAPT=0` is mandatory — the adaptive default disables the head for
whole requests and **halves decode**. Depth 2 all-round, 3 for code.
- **Batch XOR speculation was engine-version-specific:** halogen 0.1.4 lost the drafter above one slot
(41 → 14 t/s); flash-server runs two slots *with* the head. On the ROCmFPX/llama.cpp line
`--spec-draft-n-max` defaults to **16** and must be pinned to 4 — the measured knee on the dense
family, with the draft head offloaded (`--spec-draft-ngl 999`) and ngram never layered under it.
- **Prompt cache is free here and was manual on the last engine:** flash-server ships it on and its
auto-sizing fits this box's ledger (no pin) — a 100k follow-up re-prefills in ~2 s instead of ~88 s
cold (env, 2026-10-17), and the 2026-10-18 A/B measured 81.5 s cold against 0.3 s warm at 101,987
prompt tokens. halogen 0.1.4's auto-sizing refused here and re-prefilled every 79–81k turn in
85–180 s until `HALOGEN_CACHE_MB=16384` was pinned. What the pool takes is what the table loses, so
re-measure the warm ratio after any pool change.
- **400-zone law:** the harness window MUST sit ~45k under the engine ceiling (`contextWindow:
155648` = 200,704 − 45,056) because the harness clamps `max_tokens` to `contextWindow − est − 4096`,
its chars/4 `est` under-counts code/agent traffic by ~3 %, and compaction only fires at 85 %. The
3 % is measured, not theory: 2026-10-07 at the old 200,192 ceiling, est 152,938 vs server 157,388 —
over by 354, a 400. At the live shape compaction fires at 132,301 est ≈ 136k measured (the 180k
target of the 262,144 era is unreachable — the window itself is 155,648), and the worst-case request
is 155,648 − 4,096 + 3 % ≈ 156.2k ≤ 200,704: ~9.5× the error of margin, robust to a ~25 %
under-count. Compaction rides [[frank-zealot]] at ≈ threshold − keepRecent(20k) ≈ 112k input.
- **Admission reserves prompt + `max_tokens`:** an un-fittable request waits in arrival order, then
503s `engine_busy` for the whole queue after `QUEUE_TIMEOUT` — and the cap/timeout pair moves
together: 65536 tokens at ~10 t/s serial needs >6550 s, which is why 7200 is pinned (env, 2026-10-17).
- **Depth is the cost on this silicon:** ~8 µs per token of context per round (KV reads ~8 GB/s against
180–240 GB/s of weight streaming) — 30–50 t/s shallow, 3.3–3.8 at 79–81k, so deep-conversation roles
belong on the fp8-KV endpoint [[frank-archon]]. BF16 dense decode tops out at 13.1 t/s (≈94 % of the
roof): only speculative decode moves it.
- **The file picks the runtime, never the reverse:** v2 `.hgn` only in flash-server, the 27B `.hgn` only
in halogen, ROCmFPX types 101/102/103/105/108 in no stock loader; fork builds pin by SHA —
pre-`36e9acd` ROCmFPX loads the AGENT files then SIGSEGVs in `ggml_compute_forward_set_rows`.
- **Same filename ≠ same model:** native v2 is 66,687,678,432 B, the Swift1.5 r2 is 66,687,678,464 B.
Byte count + LFS sha256 is the only discriminator, and upstream versions sidecars in place, so equal
sizes prove nothing.
- **Companion law:** the n-gram table is REQUIRED beside the checkpoint; if the publisher ships only the
main file, hardlink the engine vendor's stock table/tower/tokenizer (a hardlink is a regular file and
passes the no-symlink gate) and referee with the engine's own tools — a mismatched table raises KL
*everywhere* and degrades retrieval, which is what NIAH + needle probes rule out.
- **Payload gates:** flat tokenizer directory, every entry a regular file (HF cache symlinks dangle
inside the mount), and the serve gate must mirror the deployer exactly or deploy goes green while the
unit crash-loops.
- Serve-script lore (this file has been bitten by all three): capture `podman images -q` into a
variable *before* grepping it — under `pipefail`, grep's early exit killed the pipeline and the
deploy refused with "no match for the pinned digest" on a box that had the image (2026-09-30); never
put a comment inside the `podman run` backslash chain — a mid-chain `#` terminates the continuation
and runc gets `--name run` (2026-10-05, exit-127 crash loop); pass only `--ulimit memlock`, since
Dankclaw's nofile hard limit is 524,288 and rootless runc cannot raise it (exit 126), and clear the
legacy `frank-dragoon` container name before starting.
- **Bench discipline (this endpoint wrote the fleet law):** stop the service for any GPU probe
(`census-frank-dragoon.sh verify` is the only GPU-free one); `gufo bench` cannot rank speculative
settings and inverts them; ≥128 generated tokens or the per-token figure is first-token noise; card
acceptance/t-s numbers are file properties, not promises — the live card's 53–57 t/s greedy decode
and ~1,640 t/s prefill at 34k measured 50.0 and 1,604 here (2026-10-17), and the 27B card's 31.7
t/s measured 27.6–28.1; health 200 does not prove your pid owns the port; there are no scratch ports
on this host — A/B through `scripts/trial-frank-dragoon.sh` drop-ins; never `pkill -f llama-server`
(frank-zealot is one), sweep by payload name.
- **Dead ends — do not re-try:** ngram under MTP or DFlash (slower at every depth; on recurrent
Gated-DeltaNet layers it also disables MTP's rollback path, −7…−15 %); DFlash across parallel slots
(shared draft ctx spuriously emits EOG); `-ot`/`-ncmoe` on a hybrid split (disables mmap, page-locks
~108 GiB and stranded the host until a power cycle); `--no-mmap` on any PLE-table file (silent cgroup
OOM-kill); forcing PLE to CPU (23 → 13.4 t/s); `HALOGEN_FA_TQ=256` (no decode win, −9.5 % prefill);
big ubatches as a decode knob (prefill-only, ub 512 fastest); Flash-Next MTP heads on the qwen4exp
loader (only `blk.48`, never the full per-layer norm set); mradermacher i1 GGUFs (NextN stripped);
vLLM quant paths on gfx1151; a custom Jinja template on gufo (no Jinja engine — send the `terse` text
as a client system message instead).
- Build lore if a fork is rebuilt: purge Ubuntu `librocwmma-dev` (shadows AMD's headers, `-I` cannot beat
it), pin `--gcc-install-dir` to GCC 15, `-DLLAMA_BUILD_WEBUI=OFF`; four HIP processes is the device
ceiling and ComfyUI is a co-tenant; `amdgpu: SVM mapping failed` spam is a symptom, not a cause.
## Contract (wire)
- Live: `/v1/models`, `/health`, `/metrics`, POST `/v1/chat/completions`; `404`: `/ping`, `/props`,
`/completion`, `/tokenize`. **No `Server-Timing` header and no `/completion` timing object** — measure
wall-clock plus `usage.prompt_tokens`.
- Thinking lands in `reasoning_content` and **counts against `max_tokens`**: a budget that dies
mid-thought returns `finish_reason: "length"` with empty `content`, so clients read both fields; size
vision probes at ≥512 tokens or the model looks blind. Budget for the base reasoning long: 1,476
thinking tokens mean on the 2026-10-17 battery, against 784 for the retired Swift1.5 finetune (×1.9)
and 905 for the censored 27B.
- `reasoning_effort` accepts only `xhigh|medium|low` (anything else is a template error — the
`models.yml` fold stays) and there is **no `"none"`**: disable thinking with `enable_thinking:false`,
which is also what the llama.cpp-era QwenSharp template wanted (`reasoning_effort:"none"` threw
there too). `preserve_reasoning` is a 400 on this engine; the llama.cpp-era key was
`chat_template_kwargs.preserve_thinking`, which decided whether `<think>` blocks stayed in recorded
assistant messages. Sampler defaults 1.0/0.95/20 are server-side, so an absent temperature is NOT
greedy (halogen 0.1.4 and w4b decoded greedily); unknown sampling fields are a 400, out-of-range
values are refused not clamped, and `response_format` is unsupported.
- Over-ceiling prompt → 400 naming the exact overage, rejected by preflight in <1 s, so probing the
ceiling is free; over-cap `max_tokens` → 400 naming the cap; `max_tokens`, `max_completion_tokens` and
`max_output_tokens` all work and must agree.
- Vision: OpenAI `image_url` parts only (top-level `images` is the frank-zealot dialect), ≥256 px or the
model answers "video", ~2,040 tokens per image, `http(s)` image URLs refused on purpose.
- The first request after a boot is a warmup artifact and may sit silent 45–60 s with 503
`engine_unresponsive` then self-recover — do not restart over it; `/health` 503 during load is loading.
Flags read once at startup, all bitwise except `HALOGEN_CK_OVERLAY`, `HALOGEN_MATMUL_TUNING_FILE`,
`HALOGEN_INDEXER_BUDGET`, `HALOGEN_COMPOSABLE_CONTEXT`.
- Deploy waits for :7770 to be free: a rootless container binds it at CREATE and exits 126, and a
leftover Quadlet beside the root unit guarantees the race.
## Files
- `scripts/frank-dragoon.env` · `scripts/serve-frank-dragoon.sh` (gates, RAM pin-gate wait, foreground
`podman run`, ledger to `/home/dank/logs/frank_dragoon.log`) · `scripts/frank-dragoon.service`
(`User=dank`, `LimitMEMLOCK=infinity`, `Restart=always`, `OOMScoreAdjust=-500`, `ExecStop=podman stop
-t 60`) → `deploy_llm_dragoon` in `scripts/deploy-remote.sh`.
- Fetchers: `download-frank-dragoon-v2abl.sh` (live); era rollbacks `-flashnext`, `-halogen`,
`-kingjones`, `-kingjones-uncensored`, `-swift`, `-unsloth`, `-q4`, `-orca`,
`download-frank-dragoon.sh`; runtime pins `build-rocmfpx.sh`, `build-gufo.sh`.
- Verification: `tests/frank_dragoon_unit.bats` (payload gates, deployer/serve parity, spec block, KV
band), `tests/llm_units.bats`, `tests/llm_secrets.bats`, `scripts/census-frank-dragoon.sh`,
`scripts/battery-frank-dragoon.py`, `scripts/stage-frank-dragoon-eval.sh`,
`scripts/trial-frank-dragoon.sh`.
- Related: [[deployment]] · [[Model Swap and Rollback]] · [[frank-zealot]] · [[frank-archon]] ·
[[dankclaw-iommu-translation-off]] · [[frank-redo-2026-08-17]] · [[Dankclaw System Specs]] ·
[[DFlash DSpark Speculative Decoding]] · [[qwen3.8-spec-matrix]] · [[qwen3.8-strix-optimizations]].
serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md
+1 -1
View File
@@ -211,7 +211,7 @@ re-checking that 35:30 and 37:28 still fail so the gate keeps discriminating.
- [[frank-medium-llamacpp-dflash2]] — the deployment this tuned, and its split
history
- [[frank-zealot-benchmark]] — the same sweep on a CUDA 3090, where a 16k prompt
- [[frank-zealot]] — the same sweep on a CUDA 3090, where a 16k prompt
at depth 0 made prefill climb all the way to ubatch 2048. Depth is the
difference: with no KV to attend over, per-pass overhead dominates and a
bigger micro-batch wins; at 64k of resident KV it does not.
serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md
+1 -1
View File
@@ -724,7 +724,7 @@ fleet.
2026-09-15, whose `--split-mode layer` measurements this service reuses
- [[Dual-R9700-Cooperation]] — why the two cards cannot take part in a
collective, and every software escape that is closed
- [[frank-zealot-qwen38-gsq-rco]] — the fleet's other llama.cpp endpoint (RTX
- [[frank-zealot]] — the fleet's other llama.cpp endpoint (RTX
3090, MTP, the image path)
- [[Dankserver System Specs]] — hardware inventory
- [[frank-medium-batch-ubatch]] — the batch/ubatch sweep that set the deployed
serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md
+2 -2
View File
@@ -25,7 +25,7 @@ timestamp: 2026-09-15
when the RTX 3090 left the box and two **AMD Radeon AI PRO R9700** cards
(Navi 48, RDNA4, `gfx1201`, 32,624 MiB each, PCIe 5.0 x16) took its place.
It replaced the retired FreeToken/CUDA `frank-small`
([[freetoken-frank-small]]), which cannot run on this hardware at all.
([[frank-zealot]]), which cannot run on this hardware at all.
Stack: **charlie12345/ROCmFPX llama.cpp** (`main`, `c49ebdb`, 2026-08-22)
serving **kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF** with the
@@ -213,7 +213,7 @@ depend on any of it.
## See Also
- [[freetoken-frank-small]] — the retired predecessor (FreeToken/CUDA on the 3090)
- [[frank-zealot]] — the retired predecessor (FreeToken/CUDA on the 3090)
- [[kingjones-q8agent-rocmfpx]] — the gfx1151 investigation of this checkpoint family
- [[Llama Cpp Runtimes]] — where the ROCmFPX fork sits among the runtimes
- [[Dankserver System Specs]] — current hardware
serverAppConfigs-docs/LLM/frank-npu.md
+2 -2
View File
@@ -25,7 +25,7 @@ timestamp: 2026-09-11
> file and serve script are staged in `serverAppConfigs/.AGENTTRASH/`
> (suffixed `.flm-npu`), and the host unit is still masked
> (`systemctl is-enabled frank-npu` → `masked`, read 2026-10-18). Port 7771
> belongs to [[frank-zealot-llama-prism-bonsai2-pq20]] today.
> belongs to [[frank-zealot]] today.
The third inference endpoint on Dankclaw: a FastFlowLM (FLM) v1.0.1 server
(`/usr/bin/flm`, dpkg `fastflowlm`) on the **XDNA2 NPU** (50 TOPS), serving the
@@ -161,4 +161,4 @@ retirement; the routes are mapped in the table above.
- [FastFlowLM](https://github.com/FastFlowLM/FastFlowLM) — FLM runtime and NPU model packs
- [[dankclaw-iommu-translation-off]] — the decision that retired this endpoint
- [[halogen-flash-server]] — the surviving image-input path
- [[frank-dragoon]] — the surviving image-input path
serverAppConfigs-docs/LLM/frank-redo-2026-08-17.md
+3 -3
View File
@@ -14,11 +14,11 @@ The pipeline-parallel split and the FastFlowLM NPU endpoint were retired.
The old scripts moved to `old/scripts/` in this repository.
> **HISTORY — the topology below is two fleet generations dead.** Today the
> fleet is three endpoints: [[frank-archon-radiance-qwen38-iq4r]]
> fleet is three endpoints: [[frank-archon]]
> (`192.168.0.235:7770`, radiance 1.0.13, 2×R9700 TP=2),
> [[frank-dragoon-flash-server-v2-native]] (`192.168.0.143:7770`, halogen
> [[frank-dragoon]] (`192.168.0.143:7770`, halogen
> flash-server 0.16.2 on the iGPU) and
> [[frank-zealot-llama-prism-bonsai2-pq20]] (`192.168.0.143:7771`, llama.cpp
> [[frank-zealot]] (`192.168.0.143:7771`, llama.cpp
> PrismML fork build 10754 on the RTX 3090 — the card that sat in Dankserver
> until 2026-09-15). No vLLM and no ROCm endpoint serves anything; there is no
> pipeline parallelism and no NPU endpoint.
serverAppConfigs-docs/LLM/frank-zealot-benchmark.md
-193
View File
@@ -1,193 +0,0 @@
---
timestamp: 2026-09-03
type: reference
title: frank-zealot Benchmark — ncmoe and Batch Tuning
description: llama-bench sweep of MoE CPU offload (ncmoe) and batch/ubatch on the RTX 3090 for Qwen3.6-35B-A3B Q6_K_P.
tags:
- benchmark
- llamacpp
- ncmoe
- ubatch
- dankserver
---
# frank-zealot Benchmark — ncmoe and Batch/Ubatch Tuning
> **HISTORY (runs 2026-08-13 and 2026-08-25, Dankserver's RTX 3090).** That
> card was pulled from Dankserver on 2026-09-15 (`lspci | grep -ci nvidia` → 0
> there on 2026-10-18 — [[Dankserver System Specs]]) and today drives
> frank-zealot on **Dankclaw**: [[frank-zealot-llama-prism-bonsai2-pq20]]
> (`Ternary-Bonsai-2-27B-PQ2_0.gguf`, 2 × 200,704 q8_0 KV, MTP off, no
> `--n-cpu-moe` — the model fits the card outright). What survived the payloads:
> decode is ubatch-invariant, CUDA prefill climbs monotonically to ubatch 2048,
> and the logical `--batch-size` is neutral — the live env still ships
> 4096/2048 (`scripts/frank-zealot.env:44-45`).
This page records a llama-bench sweep on Dankserver (RTX 3090, 24 GiB) for
the HauhauCS Qwen3.6-35B-A3B Q6_K_P model. The sweep decided the final
`--n-cpu-moe` and `--ubatch-size` values for the llama.cpp rung then served on
that card (the `frank-zealot` name dates from 2026-10-06/10-09).
Run date: 2026-08-13. Build: llama.cpp `555881ebc` (10121). CUDA, 7 threads,
`--cache-type-k q8_0 --cache-type-v f16`, `--fit-target 0` (KV auto-sized to
fill VRAM). Prompt 16384 tokens, generation 512 tokens, 2 repetitions.
## Headline Findings
### Generation does not depend on batch or ubatch
For every `ncmoe` value, generation throughput stayed flat across all
`ubatch` values from 256 to 4096. The delta between the lowest and highest
was under one token per second. Generation depends only on how many MoE
expert layers stay on the GPU as `ncmoe` gets smaller.
| `ncmoe` | Expert layers on GPU | Generation (t/s) |
| ------- | -------------------- | ---------------- |
| 40 | 0 | 40.9 |
| 38 | 2 | 42.6 |
| 36 | 4 | 44.4 |
| 35 | 5 | 45.5 |
| 34 | 6 | 46.4 |
| 32 | 8 | 48.7 |
| 30 | 10 | 51.1 |
Each pair of expert layers moved to the GPU adds roughly 1.1-2.0 t/s of
generation. Moving from `ncmoe 35` to `ncmoe 32` adds about 3.2 t/s, or 7
percent.
### Prefill and graph memory scale with ubatch
Prefill throughput climbs with `ubatch` because a prompt splits into
`prompt / ubatch` micro-batches. For `ncmoe 35` at 16384 prompt tokens:
| `ubatch` | Prefill (t/s) | Peak graph VRAM (MiB) |
| -------- | ------------- | --------------------- |
| 256 | 245 | 6838 |
| 512 | 416 | 6852 |
| 768 | 556 | 7074 |
| 1024 | 710 | 7330 |
| 1536 | 895 | 7846 |
| 2048 | 1188 | 8574 |
The logical `--batch-size` does not matter. At a fixed `ubatch`, `batch`
values of 2048, 4096, and 8192 gave identical prefill speed and peak VRAM.
## Peak VRAM by ncmoe and ubatch
Peak VRAM (llama-bench, single context) rises both when `ncmoe` drops
(more experts on GPU) and when `ubatch` rises (bigger compute graph):
| `ncmoe` | ub 512 | ub 1024 | ub 1536 | ub 2048 |
| ------- | ------ | ------- | ------- | ------- |
| 38 | 4590 | 5068 | 5584 | 6086 |
| 36 | 6098 | 6576 | 7092 | 7594 |
| 35 | 6852 | 7330 | 7846 | 8574 |
| 34 | 7606 | 8084 | 8600 | 9102 |
| 32 | 9114 | 9592 | 10108 | 10610 |
| 30 | 10622 | 11100 | 11616 | 12118 |
## VRAM Budget Model
The live service holds a fixed 700416-token KV pool (f16/f16 since
2026-08-14; refitted from 800000) plus the mmproj vision tower plus CUDA
overhead. The model below fits llama-bench peak VRAM into a 24 GiB
budget:
```
service VRAM = llama-bench peak + KV pool (13.68 GiB f16) + overhead (2.2 GiB)
```
This model reproduces the historical OOMs. Configurations whose sum
exceeds 24 GiB fail. For example, `ncmoe 32 - ubatch 2048` sums to about
24.4 GiB and fails, which matches the recorded OOMs at `ncmoe 32` and
`33`. The f16/f16 KV pool (2026-08-14) costs 20,480 bytes per cell; at
800768 cells it is 15,640 MiB and does not fit the 24 GiB card, so the
service runs 700416 cells (13,680 MiB).
## Candidate Configurations
| Configuration | Gen (t/s) | Prefill (t/s) | Headroom (MiB) | Fit |
| ------------------ | --------- | ------------- | -------------- | --- |
| ncmoe 35, ub 2048 | 45.5 | 1188 | 2035 | OK |
| ncmoe 34, ub 2048 | 46.4 | 1221 | 1507 | OK |
| ncmoe 34, ub 1536 | 46.5 | 914 | 2009 | OK |
| ncmoe 32, ub 1024 | 48.7 | 766 | 1017 | OK |
| ncmoe 32, ub 768 | 48.8 | 600 | 1273 | OK |
| ncmoe 30, ub 1536 | 51.2 | 1009 | under | OOM |
## Chosen State
> **Live drift (2026-08-22 audit):** the deployed `frank-zealot` now runs `--n-cpu-moe 38`, `--batch-size 4096`, `--ubatch-size 2048` (ncmoe raised from 36 for stability; batch lowered). The values below are the original benchmark recommendation.
Prefill speed is the priority, so the state keeps the top prefill tier and
moves one expert layer to CPU for stability:
```
--n-cpu-moe 36 # 4 expert layers on GPU
--batch-size 8192
--ubatch-size 3072
```
This gives prefill about 1427 t/s at 16384 prompt tokens, generation about
44.4 t/s, and about 2.0 GiB of headroom at the 700416-token f16/f16 pool
(2026-08-14 refit from 800000; the dual-f16 pool costs 13.68 GiB and
removes the context-proportional flash-attn F16-conversion workspace that
caused the cuBLAS crash class).
`ncmoe 36` came from a live crash rather than this sweep. At `ncmoe 35` with
`ubatch 3072`, a big prefill burst failed a cuBLAS call under memory
pressure (a `cudaErrorInvalidValue` in `ggml_cuda_mul_mat_cublas`), which
killed the service. Raising `ncmoe` one step frees about 0.75 GiB of peak
VRAM (the grid shows `ncmoe 36 - ubatch 3072` peak 8606 MiB versus 9360 MiB
at `ncmoe 35`) for about 2 percent less prefill and 2 percent less decode.
If a crash still occurs under load, set `GGML_CUDA_FORCE_MMQ=1` to bypass
the cuBLAS path for quantized matmuls, or drop `ubatch`, or shrink the
context pool.
If decode speed matters more than prefill, swap to `ncmoe 32 - ubatch 1024`
(generation about 48.7 t/s, prefill about 766 t/s, about 1.0 GiB headroom).
## Ornith Q6_K — 8K `ubatch` Sweep Without Speculation
Run date: 2026-08-25. This controlled sweep used the active Ornith Q6_K model
and hybrid expert placement, but explicitly disabled speculative decoding with
`--spec-type none`. It used one slot, an 8192-token context, a fixed logical
`--batch-size 4096`, prompt caching disabled, and one unique approximately
8K-token prompt per point. The production service was stopped during the
sweep and automatically restarted afterward.
| `ubatch` | Prompt tokens/s | Generation tokens/s | CUDA compute reservation |
|---:|---:|---:|---:|
| 1024 | 1944 | 53.91 | 248 MiB |
| 1280 | 2030 | 53.87 | 248 MiB |
| 1536 | 2190 | 53.66 | 248 MiB |
| 1792 | 2203 | 53.69 | 248 MiB |
| 2048 | 2423 | 53.80 | 248 MiB |
Prefill improves through `ubatch=2048`, while generation is effectively flat.
The 8K prompt sweep therefore favors `ubatch=2048` as the balanced point; a
larger value was not tested in this corrected no-spec run.
The same workload with `draft-mtp` enabled produced approximately 26.8–35.2
generation tokens/s at these `ubatch` values, with variable draft acceptance.
The no-spec path produced approximately 53.7–53.9 generation tokens/s. This
means MTP was a net throughput loss for this workload and configuration,
despite accepting draft tokens. The production service remains on MTP until a
separate decision is made; acceptance count alone is not sufficient to select
speculation.
## Limitations
llama-bench measures raw decode and prefill. It does not run speculative
decoding, parallel slots, or the mmproj vision tower. The live service adds
those on top. The VRAM model uses a fixed 2.2 GiB overhead estimate; the
real overhead varies a little with load. The Ornith sweep above used the
OpenAI-compatible server API rather than `llama-bench`, and its throughput
includes the server's timing instrumentation.
## See Also
- [[Dankserver System Specs]]
- [[deployment]]
- [[DFlash DSpark Speculative Decoding]]
serverAppConfigs-docs/LLM/frank-zealot-gemma4-12b.md
-134
View File
@@ -1,134 +0,0 @@
---
type: reference
title: "frank-zealot on Dankclaw — llama.cpp, Gemma4-12B QAT Q4_K_M"
description: "The RETIRED frank-zealot runbook (2026-09-17 → 10-01) — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves."
tags:
- llm
- frank-zealot
- dankclaw
- llama.cpp
- rtx3090
- gemma
- mtp
- vision
timestamp: 2026-09-17
---
# frank-zealot — llama.cpp Gemma4-12B QAT Q4_K_M (MTP + ngram-mod)
> **RETIRED 2026-09-17 → 2026-10-01** (AGENTS.md Runbook Index). It replaced
> [[frank-zealot-qwen38-gsq-rco]] (ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S) and
> [[frank-zealot-vllm-gemma4]] (the same-day vLLM experiment).
> Same port (7771), same alias (`frank-small` — this endpoint only carried the
> `frank-zealot` name from 2026-10-06/10-09), same client contract surface.
> :7771 today runs [[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML
> fork build 10754, `Ternary-Bonsai-2-27B-PQ2_0.gguf`, 2 × 200,704 q8_0 KV, MTP
> off); the audio input this page documents left the fleet with this payload.
## Identity
- **Endpoint:** `frank-small` at `192.168.0.143:7771` (Dankclaw) —
`git show e23df8b:scripts/frank-small.env` → `ALIAS=frank-small`.
- **Runtime:** mainline llama.cpp master `37b53fd`, CUDA build for sm_86 with
`-DGGML_CUDA_FA_ALL_QUANTS=ON`
(`/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server`).
The FA_ALL_QUANTS flag is required: without it, q8_0 KV + flash attention
collapses prefill ~12x on CUDA. Host check 2026-10-18: that binary reports
`0.4.1-dev (build 1, commit 37b53fd)` and its `CMakeCache.txt` reads
`GGML_CUDA_FA_ALL_QUANTS:BOOL=ON`.
- **Service shape:** root systemd unit `scripts/frank-zealot.service`
(`User=dank`), launched by `scripts/serve-frank-zealot.sh` with tunables from
`scripts/frank-zealot.env`.
## Shape
- **Model:** `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` at
Q4_K_M (6.87 GiB). Architecture `gemma4`: 48 layers, MHA with 16 kv heads
x 512 head_dim, rope freq_base 1e6, final logit softcapping 30, per-layer
token embeddings (PLE) inside the file, and a sliding-window attention
pattern (`attention.sliding_window = 1024`) — most layers cache only a
1024-position window, which is why the KV pool is a fraction of the
full-ctx x all-layers number.
- **Context:** four FULL 262,144-token slots (`CTX_SIZE=1048576`,
`PARALLEL=4`). The explicit `CTX_SIZE` bypasses `--fit` on purpose: `--fit`
maximises the TOTAL pool, which would shrink each slot below the full window
once parallel > 1. The 4-slot boot idles at 22,346 MiB; each extra
full-context slot costs 3,450 MiB, and a fifth projects to ~25.8 GiB — past
the card. 4 is the measured max.
- **Placement:** the whole model in VRAM (`--gpu-layers 999`); the host CPU
only tokenizes and samples (its CPU belongs to the halogen engine serving
frank-dragoon).
## Spec method
The MTP head ships as a **separate** file (`mtp-gemma-4-12B-it.gguf`, arch
`gemma4-assistant`, 4 nextn layers, 0.24 GiB), loaded via `--model-draft`
with `--spec-type draft-mtp`. The same `--spec-type` value chains the
draftless **ngram-mod** matcher, which attaches unconditionally; two
context-backed drafters cannot share one process, so `draft-mtp,ngram-mod` is
the only legal shape that carries both. Draft depth 3 + poll 50 (measured:
depth 4 + poll 100 decoded WORSE at long context — the longer verification
round costs more attention re-read than the extra accepted token earns).
## KV
**q8_0** (`--cache-type-k q8_0 --cache-type-v q8_0`), with flash attention
(`--flash-attn on`, required for quantized KV). CPU-side cache capped at 2048
MiB (`CACHE_RAM=2048`) to give back ~6 GiB of LPDDR — Gemma4 runs fully on the
3090, so the CPU cache is not the hot path, and it is not a DeltaNet model so
the >12.8 GiB floor does not apply.
## Vision + audio
`--mmproj` loads the BF16 projector
(`mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf`, 11 tensors,
52M params), which also carries an audio encoder. The fleet's image path
lives here. Audio works: `input_audio` content parts transcribe verbatim
(verified 2026-09-17 with a 16k mono WAV). Fleet-wide audio ended with this
payload: no fleet endpoint takes audio input today.
## Payload
The main GGUF, the BF16 mmproj, and the MTP sidecar lived under
`/home/dank/models/Gemma4-12B-QAT`; they were staged to
`.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/` on 2026-10-02 and
that stage dir is gone as of 2026-10-18, so the payload needs a re-fetch. The serve script refuses to start if any
is missing (a missing file is an unfinished download, and the unit's restart
policy would otherwise turn it into a crash loop that reads like a runtime
fault).
## Client contract
- **Alias:** `frank-small` (the `id` at `/v1/models`).
- **Reasoning:** extracted to `reasoning_content` (`--reasoning-format
deepseek`, unlimited budget).
- **Tools:** supported (`--jinja` is pinned even though this build defaults
it on, so a future build that flips the default cannot silently take tools
away).
- **Vision + audio:** both supported in this era; audio is gone fleet-wide
today (`input: [text, image]` — `~/.omp/agent/models.yml:214`).
- **Sampling:** greedy by default (fleet contract); a client that sends a
temperature gets its own, and the harness applies per-model overrides from
`models.yml`.
## Measured (max length)
At 254k-token prefill / 253k-ctx decode, 4 slots, q8_0 KV, ubatch 1024:
pp ~965 t/s, tg 257–302 t/s, idle 23,300 MiB of 24,576. Larger ubatches
(2048) OOM the MTP draft context's pp compute buffer and must not be tried.
## Rollback
Payload and runtime for [[frank-zealot-qwen38-gsq-rco]] (ISTA-DASLab
Qwen3.8-27B-GSQ-RCO IQ3_S) are **not on disk any more** (the 2026-10-18 census
of `/home/dank/models/` has no GSQ-RCO directory and the era fetch script is
not in the tree) — that rollback re-downloads from ISTA-DASLab.
[[frank-zealot-vllm-gemma4]] is the same-day vLLM experiment; its weights are
gone too, though its derived image is still in rootless storage
(`localhost/vllm/vllm-openai:frank-small`, id `8dad46d79c46`).
# Citations
- Model + mmproj + MTP sidecar: https://huggingface.co/HauhauCS
- llama.cpp: https://github.com/ggml-org/llama.cpp
- Serve script: `scripts/serve-frank-zealot.sh`; tunables: `scripts/frank-zealot.env`
serverAppConfigs-docs/LLM/frank-zealot-llama-gemma4-q4km-200704.md
-116
View File
@@ -1,116 +0,0 @@
# frank-zealot — llama.cpp CUDA, Gemma4-26B-A4B Q4_K_M, 1×200,704 (Dankclaw)
**RETIRED 2026-10-07** — same day it went live (AGENTS.md Runbook Index).
Superseded by
[[frank-zealot-llama-qwen35-9b-q6k-2x200704]] (Qwen3.5-9B Q6_K at
2 × 200,704 with exact KV); :7771 today runs
[[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML fork build 10754,
`Ternary-Bonsai-2-27B-PQ2_0.gguf`, 2 × 200,704 q8_0 KV, MTP off). The payload
stays at `/home/dank/models/Gemma4-26B-A4B/` as the rollback (all three files
present in the 2026-10-18 host census). This page keeps the
Gemma4 KV-billing law (the `attention_k_eq_v` pair-billing measurement) and
the drafter-cost boots, both of which outlive the model.
## Payload
|File|Path|Bytes|
|---|---|---|
|Main (Q4_K_M)|`/home/dank/models/Gemma4-26B-A4B/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf`|16,796,015,520 on disk; the engine reports 16,780,192,888|
|Vision (mmproj BF16)|`/home/dank/models/Gemma4-26B-A4B/mmproj-Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf`|1,194,827,776 B on disk (2026-10-18) = 1.11 GiB|
|MTP assistant (on disk, NOT loaded)|`/home/dank/models/Gemma4-26B-A4B/mtp-gemma-4-26B-A4B-it.gguf`|251,937,728 B on disk (2026-10-18) = 0.23 GiB|
Repo: `HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP`
(base `google/gemma-4-26B-A4B-it`). Downloaded 2026-10-07 by plain
`curl -fL` (no token needed, ungated).
## Shape
```
--ctx-size 200704 --parallel 1 # ONE conversation, Frank's 200k cap
--flash-attn on
--cache-type-k f16 --cache-type-v q8_0 # exact K, FP8-class V (q8_0)
--batch-size 4096 --ubatch-size 1024
--n-gpu-layers 99
--log-verbosity 4 # debug: KV layout, buffers, graph capture
--spec-type none # MTP OFF (see below)
--mmproj <BF16 projector>
--jinja --reasoning auto --reasoning-format deepseek
```
Measured boot at **q8_0 K + q8_0 V, ubatch 2048, MTP off** (2026-10-07
morning): **21,890 MiB used / 2,235 MiB free** (`nvidia-smi`). With MTP ON
(the assistant loaded) the same shape was **23,806 / 319** — the drafter's
context costs ~1.9 GiB because its KV is built against the target's full
window ([[Llama Cpp Serving Sizing]]).
**Deployed shape (2026-10-07 evening, Frank): K = f16, ubatch 1024, verbosity
4. Measured boot: 21,868 MiB used / 2,257 MiB free** — i.e. the f16-K raise
(+2,161 MiB of KV, see the accounting below) was paid for by the ubatch cut
(2048 → 1024 returns ~616 MiB of compute buffer) plus slot-geometry variance.
Exact K came essentially free. The verbosity-4 ledger (journal, `llama_kv_cache`
lines): model 16,002.91 MiB (CUDA0) + 577.50 MiB (CPU-mapped), **KV 3,001.25
MiB (5 full layers) + 306.25 MiB (25 SWA rings)**, compute buffers 926.39 +
150.63 MiB, output buffer 1.00 MiB.
**Measured decode: 139.9 tok/s** (`/completion` timings, temp 0, 256 tokens,
deployed shape) — the ~4 B-active MoE is at home on the 3090. The model
thinks: a 256-token budget on a short prompt can be consumed entirely by
`reasoning_content` with empty `content` and `finish_reason: length`, so
harness requests must budget for thinking (same law as the uncensored Swift
payload). Vision probe on this build: solid-red 512 px PNG via the OpenAI
`image_url` part → `Red`, `prompt_tokens` 151 (the image reached the model).
Decode is expected ubatch-invariant (Swift-era 3090 law; the 139.9 t/s above
is at ubatch 1024 — one rung measured, not a ladder).
## Architecture facts
Gemma4 MoE hybrid: 30 layers, 128 experts (8 active), n_embd 2816, 262,144
native window, n_vocab 262,144. Attention is hybrid per
`gemma4.attention.sliding_window_pattern` (read from the file at load): **25
sliding-window layers (window 1024, `n_head_kv = 8`) + 5 full-attention layers
(indices 5, 11, 17, 23, 29, `n_head_kv = 2`)**, head_dim 256 for K and V in
both classes. The KV pool is TWO caches: the full-context cache scales with
the 5 layers at the full 200,704 window; the SWA cache is a fixed 1024-token
ring per sliding layer.
**The rate law (measured, exact):** bytes/token = `n_kv_heads × 256 × 6.125 B`
per layer at f16 K + q8_0 V — `attention_k_eq_v` does **NOT** shrink the
allocation: `Vcur = Kcur` is a compute-graph mapping, the cache reserves K and
V slots and, with `k_eq_v`, reserves the pair **twice** (6.125 = 2 × (2 B f16 +
1.0625 B q8_0)). Verified against the boot: 5 × 2 × 256 × 6.125 = 15,680
B/token × 200,704 = **3,001.25 MiB** ✓ and 25 × 1024 × 8 × 256 × 6.125 =
**306.25 MiB** ✓. At q8_0 K + q8_0 V the rate is 2.125 B/elem: the same shape
costs 1,041 + 106 = **1,147 MiB**, so the f16-K raise is **+2,161 MiB**. The
`--cache-type-v` flag is honoured, not inert.
The llama.cpp master build at commit `5fc4f3c` supports the arch
(`src/models/gemma4.cpp` and `LLM_ARCH_GEMMA4 = "gemma4"` — both present in
`/home/dank/GitPackages/llama-master` as read on 2026-10-18, `src/llama-arch.cpp:59`). Note: `grep -c`
arch names on the packed binary returns 0 for EVERY arch (including the
`qwen3_5` it serves) — check the source tree, not the binary.
## MTP law
MTP for Gemma4 is a SEPARATE file (`mtp-gemma-4-26B-A4B-it.gguf`, arch
`gemma4-assistant`), loaded via `--spec-draft-model` beside
`--spec-type draft-mtp` — unlike Swift, whose MTP head ships inside the main
GGUF. The serve script gained `SPEC_DRAFT_FILE` for this. 2026-10-07 (Frank):
**MTP disabled** — `SPEC_TYPE=none` strips both spec flags (flags move as a
set). The assistant payload stays on disk; setting `SPEC_TYPE=draft-mtp` +
`SPEC_DRAFT_FILE` re-enables it at the cost of ~1.9 GiB (boots at 319 MiB
free).
## Verification
- `curl http://192.168.0.143:7771/health` → `{"status":"ok"}`
- `/v1/models` reports `n_ctx: 200704`, `ftype: Q4_K - Medium`
- Vision probe: solid-color PNG ≥ 256 px via OpenAI `image_url` part,
expect the exact color word.
## Rollback
Revert to the Swift trio: `MODEL_FILE=/home/dank/models/Swift15-UncIQ3XXS/…`,
`CTX_SIZE=229376`, `PARALLEL=2`, in-file `SPEC_TYPE=draft-mtp` — the Swift
payload remains on disk. See
[[frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688]].
serverAppConfigs-docs/LLM/frank-zealot-llama-mimo-q5km.md
-182
View File
@@ -1,182 +0,0 @@
---
type: reference
title: "frank-zealot on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q5_K_M GGUF (retired)"
description: "The retired 2026-10-02/03 frank-zealot runbook — the Q5_K_M rung at two wide unquantized-KV slots on the RTX 3090, superseded 2026-10-03 by the Q6_K file. Kept as the rollback reference: it carries the measured boot-pair table that prices every pool shape on this card, the slot-window contract, and the misbehaves tree."
tags:
- llm
- frank-zealot
- dankclaw
- llama-cpp
- rtx3090
- mimo
- gguf
- kv-cache
timestamp: 2026-10-03
---
# frank-zealot — llama.cpp MiMo-V2.6 Q5_K_M (RTX 3090, retired)
> **RETIRED 2026-10-02 → 2026-10-03** — superseded by
> [[frank-zealot-llama-mimo-q6k-2x200704]]: the ~3.5 GiB this shape left free
> bought the **Q6_K quant rung** (same runtime, same 2 × 200,704 pool, same
> f16 KV), which measures 2,849 MiB free and 94.44 tok/s — 10 % slower
> decode is the measured price of the higher rung on this bandwidth-bound
> card. This file remains the rollback reference and the owner of the
> measured boot-pair table. :7771 today runs
> the PrismML 27B era of [[frank-zealot-llama-prism-bonsai2-pq20]] (build 10754,
> 2 × 200,704 q8_0 KV, MTP off) — retired 2026-10-17 → 10-18 as a rollback
> target; the active row today is build **10777** on
> `Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf` at 2 × 262,144 exact f16 KV, MTP off by
> measurement (AGENTS.md Runbook Index). The Q5_K_M
> payload is still on disk (`stat -c %s` 2026-10-18 → 6876124704).
> Its own history: one quant step down from [[frank-zealot-llama-mimo-q6k]]
> (2026-10-01), resized twice on 2026-10-02 (three contexts with vision, then
> the 2–3 GiB-band pool), and on 2026-10-03 morning reshaped to **two
> conversations of 200,704 tokens with the KV VALUES unquantized** —
> *"2 streams with 200700 tokens context or more… if we have extra space on
> the 3090, unquantize the kv."* That shape measured 20,530 used / 4,046
> apparent free (the true `nvidia-smi` free was 3,571 — see *Shape*).
## Identity
|Field|Value|
|---|---|
|Endpoint|`192.168.0.143:7771`, served alias `frank-small` in this era (`git show e5893dc:scripts/frank-small.env` → `ALIAS=frank-small`; `frank-zealot` only from 2026-10-06/10-09)|
|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server` — unchanged across the quant swap|
|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q5_K_M**, 6,876,124,704 B (6.40 GiB), sha256 `04cf46f2b5584ef922697d6c063f94879434fb86c467f1a09e52daf0cddaf35f` + the repo's `mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf`, 918,166,048 B, sha256 `ff348f3180a63188aa7285db85f550fe38acb61dd013c599eb8bad08d2cc2576` — both at `/home/dank/models/MiMo-Q5KM/`, fetched and byte- and sha256-verified by `scripts/download-frank-zealot.sh`|
|Unit|root `frank-zealot.service` (repo `scripts/frank-zealot.service`, `User=dank`, relinked), env `scripts/frank-zealot.env`, serve `scripts/serve-frank-zealot.sh`|
|Repo pins|`tests/frank_zealot_unit.bats` (stub-argv seam + the VRAM arithmetic), `deploy_llm_zealot` gates in `scripts/deploy-remote.sh`|
## Shape and the KV contract
`--ctx-size 401408 --parallel 2` — llama.cpp divides the pool across slots,
so each conversation gets `n_ctx_slot = 200704` (boot log 2026-10-03:
`n_slots = 2, n_ctx_slot = 200704, kv_unified = 'false'`). `CTX_SIZE` MUST
stay divisible by `PARALLEL`; the bats seam enforces it.
The KV contract is Frank's, as of 2026-10-03: **both K and V unquantized
(f16)** — `--cache-type-k f16 --cache-type-v f16`. The asymmetric contract it
replaced (f16 K + `q8_0` V) had existed to make three slots fit; with two
slots the exact-value budget opens, and 4 KiB/token × 8 layers × 2 of extra
V traffic is the measured ~2 % decode tax. The serve script still passes both
flags explicitly — the asymmetric-era law that a quantized V requires
`--flash-attn on` stays pinned by the seam.
**Pool sizes on this stack come from boots, not formulas.** The compute
buffers, CUDA graphs and vision tower all bill separately, so every shape
change is priced from measured rows:
|Shape|KV dtype (K+V)|Projector|`nvidia-smi` used|Non-KV (weights + buffers + projector)|
|---|---|---|---|---|
|2 × 262,144 (pool 524,288)|f16 + q8_0|none|20,190 MiB|**7,390 MiB**|
|3 × 196,608 (pool 589,824)|f16 + q8_0|f16|22,972 MiB|**8,859 MiB**|
|3 × 180,224 (pool 540,672)|f16 + q8_0|f16|21,684 MiB|~8,419 MiB|
|**2 × 200,704 (pool 401,408)**|**f16 + f16**|f16|**20,530 MiB**|**~7,986 MiB**|
The 32 KiB/token full-attention KV price (8 layers × 4 KV heads × 256 dim ×
2 B × 2 halves) prices the deployed pool at 401,408 × 32 KiB = **12,544 MiB
of KV**; the card is 24,576 MiB, so the shape leaves **4,046 MiB free**
(measured steady-state — the KV pool is allocated whole at boot, so this is
the ceiling). The headroom above the 2–3 GiB floor is deliberate slack, not
an invitation: another 32,768 tokens of pool (1 GiB) would trade the last
safety margin for window the harness compacts before reaching anyway.
**Why 2 × 262,144 with f16 V does not fit either.** The native window pair is
524,288 tokens = 16,384 MiB of KV plus ~7,986 non-KV = 24.4 GiB: under the
floor by a hair. 200,704 is the largest 4,096-aligned slice that keeps the
measured band.
## Measured on this shape (2026-10-03, 2 × 200,704 + vision, f16 KV)
|Check|Result|
|---|---|
|Boot to healthy|~5 s, `n_slots = 2`, `n_ctx_slot = 200704`, `loaded multimodal model` logged|
|VRAM|**20,530 MiB used / 4,046 MiB free** of 24,576 (`nvidia-smi`, steady state)|
|Decode, temp 0, no thinking|**104.84 tok/s** (128-token completion at ~500-token depth; the 3 × 180,224 q8_0-V shape measured 106.8 — ~2 % is the honest price of exact V)|
|Prefill|1,650 t/s on the same shallow 506-token probe (rates are depth- and prompt-dependent; not comparable across shapes)|
|Batch geometry|`--batch-size 2048 --ubatch-size 1024` — ubatch 2048 failed the scratch boot; 1024 boots clean|
|Vision|projector loads at boot (`loaded multimodal model`); red-PNG probe passed on 2026-10-02. Caveat discovered on 2026-10-03: probes must use a >= 256 px image — a 10x10 px PNG undersupplies image tokens and the model hallucinates ("it's a video")|
|Tools / thinking|unchanged by the reslice: native `tool_calls`, thinking in `reasoning_content`|
## What the slot geometry costs
The per-slot window is the whole price, in both directions:
- **A prompt over 200,704 tokens is a hard 400**, so `~/.omp/agent/models.yml`
sets `contextWindow: 200704` for this endpoint — the harness compacts before
it reaches the ceiling instead of discovering it.
- **frank-dragoon's tallest pre-compaction prompt now fits here.** Compaction
for frank-dragoon sessions runs on this endpoint (`compactionModel` in
`models.yml`, still true 2026-10-18 at `~/.omp/agent/models.yml:107`); frank-dragoon compacts at 70 % of 262,144 = 183,501 tokens,
which did **not** fit the old 180,224 slot and fits 200,704 with 17,163
tokens to spare (2026-10-18: frank-dragoon's `contextWindow` is 155,648 —
`~/.omp/agent/models.yml:131` — so its compaction input is smaller still).
The 2026-10-02 window trim had restored the fit for
frank-archon's smaller 140,492-token trigger; the 2026-10-03 reslice widens
the fit to frank-dragoon itself. Nothing routes compaction automatically —
the point is the ceiling no longer forbids it.
- Two simultaneous generations are the slot ceiling; a third request queues
behind a released slot (the mechanism measured on the 3-slot shape).
## Client contract
- Thinking arrives in **`reasoning_content`** (`--reasoning-format deepseek`),
the field `models.yml` reads for this endpoint.
- **Thinking is off unless the request asks**: the MiMo GGUF template defaults
`enable_thinking` false, so `models.yml` ships
`chat_template_kwargs: {enable_thinking: true, preserve_thinking: true}` in
`extraBody`. A raw client that omits it gets a direct answer — measured, and
not a fault. Trivial questions produce no thinking even with the flag on.
- Tools: `--jinja` + server-side parsing, verified 2026-10-02.
- **Vision is on**, through the projector file (`--mmproj`). Use the
OpenAI-standard `image_url` content part — that path answers correctly
("Red." on a solid-red PNG). The legacy top-level `images` field is **not**
read by `/v1/chat/completions`: the same request sent that way answered
"Brown.", i.e. it generated from text alone. `/completion` takes `images`
but refuses a request without `messages` in this build, so chat is the only
vision path.
- Setting `MMPROJ_FILE=` (explicitly empty) in `scripts/frank-zealot.env` drops
`--mmproj` and returns ~1.4 GiB of the card to KV — the documented text-only
opt-out. `MMPROJ_FILE` pointing at a file that does not exist is a boot
refusal, never a silent fallback to blindness.
## No spec decode
The GGUF has no MTP tensors (the same fact that made SGLang's NEXTN a 0.00
ghost), and the cross-model DFlash draft
(`/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf` — no longer on
disk: the 2026-10-18 census of `/home/dank/models/` has no `DFlash-Qwen35/`) is an
optimization candidate, not a default — acceptance has to be measured on this
traffic first. The bats seam fails if a draft flag ships unannounced.
## Rollback
|Target|How|
|---|---|
|Previous shape: 3 × 180,224 + q8_0 V (2026-10-02)|`scripts/frank-zealot.env` `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 MiB used, 2,441 free) and `contextWindow: 180224` in `models.yml` — choose it when three simultaneous conversations matter more than per-slot window or exact V|
|Q6_K, same runtime|`scripts/frank-zealot.env` `MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + `MMPROJ_FILE` in that dir — the deployed rung **2026-10-03 → 10-05** ([[frank-zealot-llama-mimo-q6k-2x200704]]): at 2 × 200,704 it fits with 2,849 MiB free, no pool cut needed (the older "+1.4 GiB, drop 49,152 tokens" estimate held only for the 3-slot shapes)|
|Two 262,144 windows|`CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=q8_0` (measured 20,190 MiB used, 4,386 free **with q8_0 V** — with f16 V the same shape is 16,384 MiB of KV + ~7,986 non-KV and does not fit) and `contextWindow: 262144` in `models.yml` — choose it when a single conversation needs the native window|
|Vision off|`MMPROJ_FILE=` in the env file; the deployer gate then has to be satisfied too, so point `LLM_ZEALOT_MMPROJ` at any real projector file or drop the gate in the same commit|
|SGLang + INT8|[[frank-zealot-sglang-mimo-v26]] — venv and old unit/env/serve trio still in place; the INT8 payload was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 and **that stage dir no longer exists** (`ls /home/dank/.AGENTTRASH/`, 2026-10-18), so re-fetch with `old/download-frank-zealot-sglang-int8.sh`|
## When it misbehaves
- Port 7771 dark: `journalctl -u frank-zealot -n 50`; the serve script exits 1
naming the missing file if the binary or GGUF moved.
- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM node
is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
- Requests queue behind each other: two concurrent long generations are the
slot ceiling by contract (2026-10-03 shape); a third waits for a slot to
free (the mechanism measured on the 3-slot shape: 1.7 s for three, 3.0 s for
the fourth).
- `400 … exceeds the available context size (200704 tokens)`: the conversation
outgrew ONE slot. This is not pool exhaustion — the pool is 401,408 tokens
and the per-slot window is the binding limit. Shorten the prompt, compact,
or roll back to the 262,144 shape above.
- Vision answers a color it never saw, or claims the image "is a video":
either the client sent `images` instead of an `image_url` content part, or
the probe image is too small to supply real image tokens (use >= 256 px;
see *Client contract*).
- Slow decode after a CTX or quant change: check `nvidia-smi` — pushing KV out
of VRAM into unified memory halves the rate silently.
serverAppConfigs-docs/LLM/frank-zealot-llama-mimo-q6k-2x200704.md
-163
View File
@@ -1,163 +0,0 @@
---
type: reference
title: "frank-zealot on Dankclaw — llama.cpp, MiMo-V2.6 Q6_K, 2 x 200704, f16 KV (RETIRED)"
description: "The RETIRED frank-zealot runbook (2026-10-03 → 10-05 evening, rollback target) — llama.cpp master (CUDA sm_86) serving bartowski's MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF on the RTX 3090: two conversations of 200,704 tokens with fully unquantized f16 KV, vision through the bundled projector, and the boot-pair arithmetic that prices every pool change. Superseded by the Swift-1.5-27B IQ3_S-mtp swap."
tags:
- llm
- frank-zealot
- dankclaw
- llama-cpp
- gguf
- rtx3090
timestamp: 2026-10-03
---
# frank-zealot — llama.cpp MiMo Q6_K, 2 × 200,704, f16 KV
> **RETIRED 2026-10-05 (evening).** Superseded by
> [[frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688]] — the Swift-1.5-27B
> RCO-IQ3_XXS-mtp swap (uncensored, MTP drafted). :7771 today runs
> [[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML fork build 10754,
> `Ternary-Bonsai-2-27B-PQ2_0.gguf`, 2 × 200,704 q8_0 KV, MTP off). Two facts for the trail:
> committed earlier the same day (77decb5..380aafa) was **never deployed** —
> THIS MiMo shape was the live payload when Swift cut over; (2) the KV
> contract Frank set here (f16 K + f16 V, "no further quantization on this
> endpoint") was superseded the same evening by the Swift swap's explicit
> q8_0/q8_0 order. This page stays as the rollback and the measured
> boot-pair record. Payload on disk (`stat -c %s`, 2026-10-18: 7793710624):
> `/home/dank/models/MiMo-Q6K/`.
## Identity
| | |
|---|---|
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
| Endpoint | `http://192.168.0.143:7771` — alias `frank-small` on the wire in this era (`git show e5893dc:scripts/frank-small.env` → `ALIAS=frank-small`; the id became `frank-secondary` 2026-10-06 and `frank-zealot` 2026-10-09), `/v1` OpenAI surface |
| Runtime | llama.cpp master, CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (f16 KV needs no flash-attn option; the disk build's `CMakeCache.txt` reads `GGML_CUDA_FA_ALL_QUANTS:BOOL=OFF` and that tree has no CUDA-side guard for the option — `ggml/src/ggml-cuda/fattn.cu:438-449`) |
| Unit | root systemd `frank-zealot.service` ([`scripts/frank-zealot.service`](../../scripts/frank-zealot.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
| Serve script | [`scripts/serve-frank-zealot.sh`](../../scripts/serve-frank-zealot.sh), tunables [`scripts/frank-zealot.env`](../../scripts/frank-zealot.env) |
| Weights | `/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` — bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a…` (qwen3_5 hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, native 262,144, no MTP tensors) |
| Projector | `…/MiMo-Q6K/mmproj-MiMo-V2.6-Distill-Qwen-9B-f16.gguf` — 918,166,048 B, sha256 `ff348f…` (hardlink of the Q5KM dir's copy, link-count-2 verified; the dir is self-contained) |
| Fetch | [`scripts/download-frank-zealot.sh`](../../scripts/download-frank-zealot.sh) — `-f`, byte count + LFS sha256 verified, `--verify-only`; both files verified against the pinned sha on the host before this deploy |
| Seam | `tests/frank_zealot_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, batch pair, projector opt-out) |
## Shape and KV contract
`--ctx-size 401408 --parallel 2` — llama.cpp divides the pool across slots,
so each conversation gets `n_ctx_slot = 200704`. `CTX_SIZE` MUST stay
divisible by `PARALLEL`; the seam enforces it. Both K and V are unquantized
(f16) — the flags stay explicit because `--flash-attn` must not be lost if V
ever goes back to `q8_0` (the serve script pins it, the seam pins the pin).
**The Q6_K rung is a weights-side spend only.** File delta +917,585,920 B
(+875 MiB) against Q5_K_M, same KV, same slot geometry, same batch pair.
Priced from the measured Q5 boot at this shape (non-KV 7,986 MiB): the boot
landed **21,276 MiB used / 2,849 MiB true free** — non-KV measured 8,732 MiB,
inside the 2–3 GiB floor, so the pool stayed at 401,408 and the
12,288-token step-down ladder was never climbed. Quote `nvidia-smi`'s `free`
field, never `total − used`: the driver holds ~450 MiB of reserve the
subtraction hides (that error once overstated this card's slack by a
half-GiB).
**Quant ceiling, evaluated and closed 2026-10-03 (Frank: stay here).**
`Q8_0` is the only rung above this one on the Hub (9,545,979,424 B = 9,104
MiB, +1,671 over Q6_K): at the live pool it would leave 1,178 MiB free —
under the floor — and the calibrated bandwidth law (it predicted this file's
94.4 t/s from Q5_K_M's 104.84 within 1 %) puts decode at ~72 t/s, a further
−24 % for the near-lossless rung. Fitting it was priced two ways (trim the
pool to 2 × 182,272 — which also breaks frank-dragoon's 183,501-token
compaction-prompt fit by 1,236 tokens — or revert V to `q8_0`), and both
were declined. Re-open only on evidence the Q6_K residual matters.
**Pool prices on this card come from boots, never formulas** — the measured
boot-pair table lives in [[frank-zealot-llama-mimo-q5km]] (*Shape and KV
contract*) and still prices every rollback shape.
## Compute-buffer geometry (2026-10-05)
`BATCH_SIZE=4096 UBATCH_SIZE=2048` — both doubled from 2048/1024, and the trade
measured rather than assumed, because the predecessor shape recorded **"ubatch
2048 failed the scratch boot"** and this card has less free space than it did.
**That failure was a different shape, and it did not reproduce here.** The
service itself is the bench (one card holds one server, so no scratch port):
the old rung was re-booted through a transient `FRANK_ZEALOT_ENV_FILE` drop-in
and reproduced the 2026-10-03 boot **byte-identically** — 21,276 MiB used /
2,849 MiB free — which makes the ubatch the only variable in the comparison.
|Probe|ubatch 1024|ubatch 2048|Δ|
|---|---|---|---|
|Prefill, 8,928 tokens|3,972 / 4,077 t/s|**4,207 / 4,212 t/s**|**+4.6 %**|
|Decode, 128 tokens at that depth|91.87 / 93.01 t/s|93.81 / 94.27 t/s|noise|
|Prefill, 279 tokens|2,675.7 t/s|2,658–2,811 t/s|noise|
|VRAM used / free|21,276 / 2,849 MiB|**21,846 / 2,279 MiB**|**−616 MiB**|
Three readings:
- **The gain only exists above the old ubatch.** A 279-token prompt is shorter
than either micro-batch, so it cannot see the knob — and it did not move.
That is the control proving the 8.9k delta is geometry, not drift.
- **Decode is flat**, as every prior sweep on this card found. Ubatch is a
prefill knob.
- **The price is 616 MiB out of a 2.8 GiB margin.** 2,279 MiB free is still
inside the 2–3 GiB band, at its bottom edge. `4096/1536` is the middle rung,
and the pool's 12,288-token (384 MiB) ladder in
[[frank-zealot-llama-mimo-q5km]] is the other way to buy room back.
**Why 2048 is right HERE while 512 was right on the R9700 sibling: the curve is
backend-specific.** The 2× R9700 layer split peaks at ubatch 512 and is 10 %
off that peak by 2048 ([[frank-medium-batch-ubatch]]); on this CUDA card the
curve climbs monotonically to 2048 (710 → 1,188 t/s on the 35B MoE sweep,
1,944 → 2,423 t/s on the Ornith 8k sweep, both with decode flat). Never carry a
ubatch verdict across backends.
The logical `--batch-size` is VRAM- and throughput-neutral — two sweeps and this
A/B agree — so 4096 costs nothing and buys nothing by itself; it is here because
that is the pair the curve was measured at.
## Measured on this shape (2026-10-03, Q6_K, 2 × 200,704, f16 KV)
|Check|Result|
|---|---|
|Boot to healthy|model + projector resident in ~2.3 s (load log `0.02.27`), `n_slots = 2`, `n_ctx_slot = 200704`, `loaded multimodal model`, health 200; deploy e2e gate 33/33|
|VRAM|**21,276 MiB used / 2,849 MiB free** of 24,576 (`nvidia-smi` `free`, steady state) — inside the 2–3 GiB floor with exact V AND the higher rung. Superseded 2026-10-05 by the batch doubling: **21,846 / 2,279** — see *Compute-buffer geometry*|
|Decode, temp 0, no thinking|**94.44 tok/s** on the identical 506-token probe that measured 104.84 on Q5_K_M — **−10 %**. Decode on this card is weight-bandwidth-bound, and Q6_K streams ~10 % more bytes per token; the quant rung buys quality with decode rate, measurably|
|Prefill|2,229.9 t/s (same shallow probe; rates are depth-dependent, not comparable across probes)|
|Vision|512×512 solid-red PNG → **`Red`** (277 prompt tokens — the image really reached the tower). See the probe-size trap in *Client contract*|
|Tools / thinking|native `tool_calls`, thinking in `reasoning_content`, thinking OFF unless `enable_thinking: true` (MiMo template default; `models.yml` ships it in `extraBody`)|
## Client contract
llama.cpp surface, live-verified on this model family: model id `frank-zealot`,
native tools via `--jinja` (`tool_calls` in the response), thinking in
**`reasoning_content`**, vision via `image_url` content parts. Two vision
traps, both measured here:
1. A top-level `images` field on `/v1/chat/completions` is silently ignored
and the model invents a color.
2. **The probe image must be big enough to supply real image tokens.** A
10×10 px solid-red PNG came back "The image is a video, so I can't see its
content" — the engine itself warns at boot that Qwen-VL wants ≥ 1024 image
tokens, and a postage-stamp image lands one patch below usable signal.
512×512 answers `Red` in 19 tokens. Always probe vision with a ≥ 256 px
image; a wrong-color answer from a tiny image is the probe's fault, not
the endpoint's.
`~/.omp/agent/models.yml` sets `contextWindow: 200704` — the PER-SLOT window,
so the harness compacts before llama.cpp answers a longer prompt with a 400.
## Rollback
|Target|How|
|---|---|
|Q5_K_M at the same shape|`scripts/frank-zealot.env` `MODEL_FILE=/home/dank/models/MiMo-Q5KM/MiMo-V2.6-Distill-Qwen-9B-Q5_K_M.gguf` (+ `MMPROJ_FILE` in that dir) and the deployer default — buys back 875 MiB (3,571 free measured) **and the 10 % decode rate** (104.84 tok/s measured). Payload sha-verified on disk|
|3 × 180,224 + q8_0 V|`CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (measured 21,684 used / 2,441 free **with Q5_K_M** — Q6_K does NOT fit that shape inside the floor; pair them explicitly)|
|SGLang + INT8|[[frank-zealot-sglang-mimo-v26]] — the 2026-10-02 staging dir `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` is **gone** (`ls /home/dank/.AGENTTRASH/` on 2026-10-18 lists only the 20261005/06/07/10/17 batches), so this arm needs `old/download-frank-zealot-sglang-int8.sh`; the `sglang-cu128` venv is still on disk|
## When it misbehaves
Same decision tree as [[frank-zealot-llama-mimo-q5km]] (*When it misbehaves*):
`No CUDA GPUs are available` ⇒ `nvidia-modprobe -u -c=0`; a 400 naming
200,704 ⇒ the conversation outgrew one slot, not the pool; slow decode after
a shape change ⇒ check `nvidia-smi` for KV pushed into unified memory; queue
behind two long generations ⇒ the slot ceiling is two, by contract.
serverAppConfigs-docs/LLM/frank-zealot-llama-mimo-q6k.md
-120
View File
@@ -1,120 +0,0 @@
---
type: reference
title: "frank-zealot on Dankclaw — llama.cpp, MiMo-V2.6-Distill-Qwen-9B Q6_K GGUF (retired)"
description: "The retired 2026-10-01 frank-zealot runbook — the first llama.cpp + GGUF shape on the RTX 3090: two native-window sessions, the asymmetric KV contract (f16 K + q8_0 V) that motivated the move off SGLang, the VRAM fit at Q6_K, and what to check when it misbehaves. Superseded the same night by the Q5_K_M payload."
tags:
- llm
- frank-zealot
- dankclaw
- llama-cpp
- rtx3090
- mimo
- gguf
- kv-cache
timestamp: 2026-10-01
---
# frank-zealot — llama.cpp MiMo-V2.6 Q6_K (RTX 3090)
> **RETIRED 2026-10-01 → 10-02** (one night): superseded by
> [[frank-zealot-llama-mimo-q5km]], the same runtime and shape one quant rung
> down. The Q6_K file stays at `/home/dank/models/MiMo-Q6K/` as the rollback
> target (on disk 2026-10-18, `stat -c %s` → 7793710624). :7771 today runs
> [[frank-zealot-llama-prism-bonsai9-mtp]] (llama.cpp PrismML fork build
> **10777**, `Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, 2 × 262,144,
> exact f16 K + f16 V, MTP off by measurement).
>
> Original order (Frank, 2026-10-01 night): move this endpoint to the GGUF
> path for **asymmetric KV** — unquantized K + q8_0 V — which only llama.cpp
> exposes per-tensor. The per-tensor lever stayed; this pairing did not — the
> live endpoint bills **exact f16 K + f16 V** — 32,768 B/token = 16,384 MiB
> over the 524,288-position pool of two 262,144 windows
> (`scripts/frank-zealot.env:57-58,80-81`).
## Identity
|Field|Value|
|---|---|
|Endpoint|`192.168.0.143:7771`, served alias `frank-small` in this era (`ALIAS` reached `frank-secondary` 2026-10-06 and `frank-zealot` 2026-10-09, commit `7c77326`)|
|Runtime|llama.cpp **master** (qwen35 model support), CUDA build `sm_86`, at `/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`|
|Payload|`bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF` **Q6_K**, 7,793,710,624 B, sha256 `ef96d05a2ddf2cbb450d1af1ac3860ec769d3575bafa70692ee5609bda3fad7d`, at `/home/dank/models/MiMo-Q6K/`|
|Unit|root `frank-zealot.service` (repo `scripts/frank-zealot.service`, `User=dank`, relinked), env `scripts/frank-zealot.env`, serve `scripts/serve-frank-zealot.sh`|
|Repo pins|`tests/frank_zealot_unit.bats` (stub-argv seam), `deploy_llm_zealot` gates in `scripts/deploy-remote.sh`|
## Shape and the KV contract
`--ctx-size 524288 --parallel 2` — llama.cpp divides the pool across slots,
so each session gets `n_ctx_slot = 262144`, the model's full native window
(boot log confirms `n_slots = 2, n_ctx_slot = 262144, kv_unified = 'false'`).
Frank's KV contract: **K unquantized (f16), V `q8_0`** —
`--cache-type-k f16 --cache-type-v q8_0`. Two hard dependencies ride on it:
- **`--flash-attn on` is mandatory**: quantized V exists only on the FA
path. Without the flag the engine falls back silently and the contract
becomes a lie. The serve script pins the trio; the bats seam pins the
script.
- The VRAM budget: attention KV lives in the 8 full-attention layers only
(4 KV heads x 256 dim). K f16 ≈ 16 KiB/token, V q8_0 ≈ 8.5 KiB/token
(34 B per 32-element block) ⇒ ~24.5 KiB/token over the 524,288-token pool
≈ **12.6 GiB**. Weights 7.8 + KV 12.6 + compute buffers/graphs ≈
**20.9 GiB measured** against 24.5 GiB — the "room for checkpoints" the
order asked for. If graph capture OOMs, halve `CTX_SIZE`; never quantize
K to fix it.
## Client contract
- Thinking arrives in **`reasoning_content`** (`--reasoning-format
deepseek`) — the same field SGLang's qwen3 parser used, so `models.yml`
`reasoningContentField` did NOT move. Live-verified 2026-10-01 22:57.
- **Thinking is off unless the request asks**: the MiMo GGUF template
defaults `enable_thinking` false; `models.yml` now ships
`chat_template_kwargs: {enable_thinking: true, preserve_thinking: true}`
in `extraBody`. A raw client that omits it gets a no-thinking answer
(measured — this is not a fault).
- Tools: `--jinja` + server-side tool parsing, verified
`tool_calls` round-trip 2026-10-01 23:0x.
- **Text-only in this era.** The GGUF carries no vision projector; the
in-checkpoint tower existed only in the SGLang safetensors, so `models.yml`
carried `input: [text]`. Vision returned with the next payloads: the endpoint
is `input: [text, image]` today (`~/.omp/agent/models.yml:214`).
## No spec decode (yet)
The GGUF has no MTP tensors (same fact that killed SGLang NEXTN), and the
cross-model DFlash draft (`/home/dank/models/DFlash-Qwen35/qwen3.5-9b-dflash-Q8_0.gguf`,
Q8_0 — that path is no longer on disk; the 2026-10-18 census of
`/home/dank/models/` has no `DFlash-Qwen35/`) is an optimization-loop candidate, not a default — `--spec-type
draft-dflash` needs acceptance measured on this traffic first. The bats
seam fails if a draft flag ships unannounced.
## Build
No ccache, CUDA 12.4 system nvcc with `--gcc-install-dir`-free gcc-13 host
compiler (nvcc 12.4 rejects gcc ≥ 14):
```
cmake -B build-cuda -S . -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON \
-DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13 \
-DLLAMA_CURL=OFF -DGGML_CCACHE=OFF
cmake --build build-cuda -j 16 --target llama-server
```
The SGLang venv (`/home/dank/GitPackages/sglang-cu128`) is still on disk; the
INT8 AutoRound payload (`/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`)
is not — it was staged to `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/`
on 2026-10-02 and that stage dir is gone as of 2026-10-18, so this rollback
re-fetches with `old/download-frank-zealot-sglang-int8.sh`: repoint the unit's serve script and env from
git history (`git show frank-dragoon-gufo-q4kxl-20261001` era is BEFORE this;
use the parent of the 2026-10-01 night swap commit).
## When it misbehaves
- Port 7771 dark: `journalctl -u frank-zealot -n 50`; the serve script
exits 1 naming the missing file if the binary or GGUF moved.
- `No CUDA GPUs are available`: the 3090 vanished from the bus or the UVM
node is missing — `nvidia-modprobe -u -c=0` (the serve script runs it).
- Requests hang at first token while `/health` is 200: both slots busy —
two concurrent long generations are the ceiling by contract.
- Slow decode after a CTX change: check `nvidia-smi` — swapping KV out of
VRAM into unified memory halves the rate silently.
serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai2-pq20.md
-226
View File
@@ -1,226 +0,0 @@
---
type: reference
title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10754), Bonsai-2-27B PQ2_0, 2 x 200704, q8_0 KV (RETIRED, rollback target)"
description: "The frank-zealot runbook of 2026-10-17 → 2026-10-18 (RETIRED — rollback target) — the PrismML fork build 10754 of llama.cpp serving prism-ml/Ternary-Bonsai-2-27B-gguf at PQ2_0 on the RTX 3090: two conversations of 200,704 tokens with q8_0 K + q8_0 V (f16 KV does not fit this card at this window), vision through the publisher's Q8_0 Qwen3-VL projector, MTP impossible (no tensors in the file), and the measured four-arm VRAM census that priced the shape. Superseded by the 9B MTP export at 3 × 262,144."
tags:
- llm
- frank-zealot
- dankclaw
- llama-cpp
- gguf
- rtx3090
timestamp: 2026-10-17
---
# frank-zealot — llama.cpp (PrismML) Bonsai-2-27B PQ2_0, 2 × 200,704, q8_0 KV
**RETIRED 2026-10-17 → 2026-10-18** (AGENTS.md Runbook Index), superseded by
[[frank-zealot-llama-prism-bonsai9-mtp]] — prism-ml/Ternary-Bonsai-2-**9B**
PQ2_0 **with the MTP head**, THREE conversations at the native 262,144 window
on fork build 10777. This era stays the **rollback target**: the payload
(`/home/dank/models/Bonsai2-PQ2_0/`) and the 10754 fork tree
(`~/GitPackages/llama-prism/`) are both still on disk. It owns the laws below —
the 34,816 B/token KV figure over 16 full-attention layers, the 25,088 MiB f16
OOM wall, and the two-boot VRAM pricing that every later shape on this card
reused.
## 2026-10-18 bench window — production unit stopped, candidate arm on 7771
As of 2026-10-18 frank-zealot sits in a deliberate, dated **bench window**
(fleet Benchmarking Policy: stop the production unit, bench candidate boots
against the service itself). The journal shows the stop at 03:38:31 and the
known llama.cpp shutdown shape — SIGTERM ignored past `TimeoutStopSec`,
`State 'stop-sigterm' timed out. Killing.` at 03:40:01 — after which the unit
reports `failed` (`is-enabled` → enabled); self-healing is merely parked
while the window runs. What answers 7771 is a **hand-booted candidate arm**
(`llama-prism-b10777/llama-server` on the 9B MTP file
`Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, alias
`frank-zealot`), not this runbook's payload: candidate-arm flags are not a
state — the pid and shape moved between two censuses ten minutes apart (one
argv read `--ctx-size 786432 --parallel 3 --spec-type draft-mtp`; a later
`/props` read `n_ctx 262144`, `speculative.types=none`); quote neither, and
expect `/v1/models` to report the candidate until the window closes. The
window's remedy is the redeploy: `./scripts/deploy.sh dankclaw --service
frank-zealot`. The production trio below is unchanged — the last PRODUCTION
boot under the unit (03:14–03:38 the same morning) printed the contract
exactly: `n_ctx = 401408`, `n_ctx_seq = 200704`, `CUDA0 KV buffer size =
13328.00 MiB`, 2 slots (journal).
**Window CLOSED 2026-10-18:** that candidate became production. The redeploy
shipped the 9B MTP export at `3 × 262,144` on fork build 10777 — see
[[frank-zealot-llama-prism-bonsai9-mtp]]; this payload is the rollback target.
## Identity
| | |
|---|---|
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB (24,123 MiB visible to the engine) |
| Endpoint | `http://192.168.0.143:7771` — served model id **`frank-zealot`** (`ALIAS`), `/v1` OpenAI surface |
| Runtime | **PrismML fork build** `/home/dank/GitPackages/llama-prism/llama-server` — vendor release, `--version` = `0.2.0-dev (build 10754, commit 2459f68b5)`, sha256 `12ae5baba033d8cd…`. NOT a preference: see *Why the fork* |
| Unit | root systemd `frank-zealot.service` ([`scripts/frank-zealot.service`](../../scripts/frank-zealot.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
| Serve script | [`scripts/serve-frank-zealot.sh`](../../scripts/serve-frank-zealot.sh), tunables [`scripts/frank-zealot.env`](../../scripts/frank-zealot.env) |
| Weights | `/home/dank/models/Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-PQ2_0.gguf` — 7,206,168,928 B, sha256 `3907dc1658db1f78a9826bf8d5bcb8dc65db0d466388937af57f2294fae62ec1` |
| Projector | `Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf` — 629,246,976 B (full sha256 pinned in the fetch script), Qwen3-VL tower; **873.10 MiB worst-case VRAM, loaded AFTER the engine's fit check** |
| Fetch | [`scripts/download-frank-zealot-bonsai2.sh`](../../scripts/download-frank-zealot-bonsai2.sh) — `-f`, byte-count + sha256 verified, idempotent, `--verify-only`, no token on the command line |
| Seam | `tests/frank_zealot_unit.bats` (argv assertions against a stub engine) |
## Why the fork — the format is the lock
Read from the file's own metadata, not the model card:
|Key|Value|
|---|---|
|`general.architecture`|`qwen35` (hybrid: Gated DeltaNet + softmax attention)|
|`qwen35.block_count`|64, `full_attention_interval` 4 ⇒ **16 full-attention + 48 DeltaNet layers**|
|Attention|24 heads / **4 KV heads** × key 256 / value 256; `rope.dimension_sections` [11,11,10,0], `freq_base` 1e7, partial rope `dimension_count` 64|
|Widths|embedding 5,120, feed-forward 17,408|
|`qwen35.ssm.*`|conv_kernel 4, state_size 128, group_count 16, **time_step_rank 48**, **inner_size 6,144**|
|Window / vocab|`context_length` 262,144; vocab 248,320 (gpt2 BPE, pre `qwen35`), eos 248046|
|`general.file_type`|**141 = PQ2_0 — 2.13 bpw (group 128)**|
|Tensor histogram|851 tensors: **`pq2_0` ×402** (custom GGML type **142**), f32 ×353, bf16 ×96|
|`prism.hadamard.*`|transform `normalized-sylvester-walsh-hadamard`, block_size 1024, axis `input-last-dimension`, sign_mode explicit, **401 named weights**, sign widths [5120, 6144, 17408], 28,672 sign values, inverse applied on `token_embd.weight`, `gdn_v_grouped` true|
|Sampler metadata|`general.sampling.temp` 1.0 / `top_p` 0.95 / `top_k` 20|
|Chat template|8,952 chars; reads `enable_thinking`, **`reasoning_effort` ∈ {`xhigh` (default), `medium`, `low` — anything else is `raise_exception`**, `preserve_thinking`, `tools`/`tool_call`, image + video content parts|
Stock llama.cpp cannot load this file: type 142 is not in its enum (`gguf-py`'s
stock `GGMLQuantizationType` raises `ValueError: 142 is not a valid…` on the
same read), and even a loader that parsed it would ignore the
`prism.hadamard.*` dequant transform the weights depend on. Same class of lock
as the ROCmFPX custom-type era. The vendor build 10754 already carries the
September upstream fixes (AVX-512 load segfault, and a `GGML_CUDA_FA_ALL_QUANTS`
-capable CUDA backend — proven by q8_0 V serving under flash attention at the
correct 34,816 B/token, below).
## VRAM census (2026-10-17, four arms, production stopped)
The engine's own `common_memory_breakdown_print` ledger + settled `nvidia-smi`,
all arms `--n-gpu-layers 99 --flash-attn on --batch-size 4096 --ubatch-size 2048`,
mmproj attached:
|Shape|KV|projected `model + context + compute`|settled used / free|verdict|
|---|---|---|---|---|
|**2 × 200,704 (pool 401,408)**|**q8_0 / q8_0**|21,439 = 6,539 + 13,627 + 1,272|**22,664 / 1,461**|**DEPLOYED**|
|2 × 200,704|f16 / f16|33,198 = 6,539 + 25,387 + 1,272|—|`cudaMalloc failed: out of memory` on the **25,088.00 MiB** KV request|
|1 × 262,144|q8_0 / q8_0|16,905 = 6,539 + 8,853 + 1,512|18,130 / 5,995|fits, but one slot serializes compaction|
|2 × 262,144|q8_0 / q8_0|≈25,758 (arithmetic, same terms)|—|does not fit (f16 analogue measured 41,118)|
|2 × 200,704|q4_0 / q4_0|15,167 = 6,539 + 7,355 + 1,272|16,392 / 7,733|the roomy fallback rung|
**KV rate law (confirmed to the MiB):** 16 full layers × 4 KV heads × 256 dim =
16,384 elements/token/cache; q8_0 bills 17,408 B per cache ⇒ **34,816 B/token**,
so the 401,408 pool = **13,328.00 MiB** — the ledger printed exactly that
(`CUDA0 KV buffer size = 13328.00 MiB`). f16 doubles it to 25,088 MiB, which is
the wall. The 48 DeltaNet layers bill a fixed recurrent state per **slot**:
context 13,627 − KV 13,328 = **299 MiB for two slots (~150 MiB each)** — pool
arithmetic alone under-prices the shape by that much.
**The projector is outside the fit check.** `[mtmd] estimated worst-case memory
usage of mmproj is 873.10 MiB` is printed before fitting but not billed in it:
settled used (22,664) − projection (21,439) ≈ 1.2 GiB is that projector plus the
CUDA context. This is the difference between "2.6 GiB free on paper" and the
real **1.4 GiB** — the same late-allocation failure class that killed the
DSpark drafter boot in the llama.cpp eras. Any resize must re-read the settled
`nvidia-smi`, never the projection.
## Measured rates (greedy, `Server-Timing`-free `/completion` `timings`)
|Probe|Result|
|---|---|
|Short, warm, single stream (128 tok)|**76.6 t/s** decode (the 9B Q6_K predecessor ran 94.4 — the ternary 27B is ~19 % slower, weight bandwidth + Hadamard dequant)|
|First request after boot|2.6 t/s prefill / 37.7 t/s decode — **warmup artifact; never report the first request** (measured twice on this payload)|
|Prefill at ~35k depth|**1,258.4 t/s** (27.8 s)|
|Prefill at 166,027-token depth|**691.7 t/s** (240.0 s), zero errors, slot intact|
|Two concurrent streams, 256 tok each|**54.7 + 54.7 = 109.4 t/s aggregate** — `--parallel 2` really batches|
|Long-context retrieval (25,960-token prompt, needle at position 180 of 300 passages)|exact: *"Frank keeps the fleet in git, only through it."*|
Production boot (the 19:44 cutover deploy, same binary/flags/file): **75.8 t/s**
single-stream decode at a short prompt — within 1 % of the trial arm, so the
ladder above carries to the shipped shape.
## Behavior contract (probes, not assumptions)
- **Reasoning** lands in `reasoning_content` (`--reasoning-format deepseek`) ✓.
- **Tool calls**: a `get_weather` request returns `finish_reason: tool_calls`
with `arguments: {"city":"Seattle"}` — valid JSON, no loop, no HTTP 500. The
fork's `KNOWN_ISSUES.md` warning about malformed/looping tool calls did not
reproduce on build 10754 at this shape.
- **Vision**: a 512 px solid-red PNG answers **"Red"** — but ONLY with a
thinking budget ≥ ~512 tokens. At `max_tokens: 64` the request returns
`finish_reason: "length"` with EMPTY `content` and half a thought in
`reasoning_content`: thinking tokens count against the budget, so a
short-budget vision probe reads as "the model is blind". Images do reach the
model (a 512 px image bills ~280 prompt tokens: 322 total vs 42 for the text
alone). Keep the ≥256 px probe rule from the earlier eras.
- **Log noise law**: `/completion` (text context) at `VERBOSITY=4` prints
`failed to parse reasoning content (response_format json_schema is currently
only supported with parse_reasoning=true), content will be stored in
'content' field` (`probe_request.cpp:789`) whenever the model closes a raw
prompt with ``. It is **cosmetic** — the full chain of thought
is in `content`. The OpenAI `/v1/chat/completions` path parses it into
`reasoning_content` correctly, which is the path the harness and every
consumer uses. Do not diagnose this as a fault, and do not drop verbosity
below 4 — the KV/buffer ledger in the journal is what VRAM forensics read.
- **Long raw-prompt completions can come back empty**: at ~26k+ tokens a bare
`/completion` request generated 0–1 tokens with no error (the model emits an
end-of-turn marker immediately, and a text context has no role frame to keep
it talking). Chat traffic is unaffected; nothing in the harness posts raw
`/completion` at depth.
## Ops notes
- **Trial-window law (cost one boot window on 2026-10-17):** a scratch env file
must use the serve script's OWN variable names — `CACHE_TYPE_K`/`CACHE_TYPE_V`,
`BIND_ADDR`, `NGL`. An invented `KV_CACHE_TYPE` is silently swallowed by the
script's `: "${CACHE_TYPE_K:=f16}"` default and the arm boots **f16**, which
for this payload is an OOM that looks like a model problem.
- **`/metrics` needs `--metrics` (fixed 2026-10-18):** the fork answers **501 Not
Implemented** without the flag, and Dankclaw's Prometheus scrapes `/metrics`, so
`up{job="frank-zealot"}` has been 0 for the whole retained metric history — the
oldest sample (2026-10-11) is already 0 — while the endpoint itself answered
`/health` 200 the entire time. `METRICS=1` in `scripts/frank-zealot.env` now
ships the flag and the seam asserts it reaches the argv. halogen serves
`/metrics` natively, which is why that target was always green.
- **Base alignment:** Bonsai-2 is the BASE (censored) Qwen3.8-27B lineage, so a
refusal is possible on a compaction or subagent request. The CRACK re-export
(`dealignai/Bonsai-2-27B-Ternary-CRACK-GGUF`, already on disk at
`/home/dank/models/Bonsai2-CRACK/Bonsai-2-27B-PQ2_0-CRACK.gguf`, sha256
`5b24ea3e…`) is a **one-line `MODEL_FILE` swap**: the header census proves
byte-identical geometry — 851 tensors, the same `{142:402, f32:353, bf16:96}`
histogram, 64 blocks, vocab 248,320, the same 8,952-char template. The VRAM
ledger does not move. If a refusal ever surfaces, that is the fix; there is
also an MTP-bearing CRACK re-export upstream
(`ManniX-ITA/…-CRACK-MTP-GGUF`, 7,657,489,920 B, +451 MB head) if decode ever
needs the drafter — untested here.
- **Compaction hub unchanged:** all three endpoints still route
`compactionModel` here; 200,704 per slot is the fleet's only window that fits
a worst-case compaction input, and `--parallel 2` keeps compaction off the
critical path of a live generation.
- **Harness side** (`~/.omp/agent/models.yml`): `contextWindow: 200704` and
`maxTokens: 16384` unchanged; `extraBody` now quotes the payload's own
1.0 / 0.95 / 20 with `preserve_thinking: true`. The `reasoningEffortMap` fold
became **load-bearing** (the template raises on any level outside
`xhigh|medium|low`) — an unmapped level is a hard 500 on the role's first
request, naming Jinja rather than the model.
- **Rollback:** revert the swap commit. The Qwen3.5-9B Q6_K payload, its mmproj
and the master CUDA build all remain on disk
(`/home/dank/models/Qwen35-9B-HauhauCS-Q6K/`,
`~/GitPackages/llama-master/build-cuda/`), and the retired era keeps its own
page: [[frank-zealot-llama-qwen35-9b-q6k-2x200704]].
- **Deployed 2026-10-17 19:44** (`deploy.sh dankclaw --service frank-zealot
--service homepage`): boot ledger 22,664 used / 1,461 free, engine fit
projected 21,439 and left 2,119 ≥ 1,872 MiB, KV 13,328.00 MiB at
`n_ctx_seq = 200,704`, `n_slots = 2`, projector loaded; live probes green
(`/v1/models` serves `frank-zealot`, exact `pong` + `reasoning_content`,
`tool_calls` `{"city":"Seattle"}`, 512 px PNG → **Red**). e2e gate 33/33.
- **Picker row is live state:** the OpenWebUI `model` row's
`meta.description` named the 9B, so it was re-keyed in the database too
(backup `webui.db.bak-20261017a`). The row was written WITHOUT restarting
`open-webui` — the cached list refreshes on its next restart, and forcing one
would have logged every family session out for a cosmetic string. See
[[OpenWebUI]] for the picker trio.
## Related
- [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — the replaced era (rollback target)
- [[frank-dragoon-flash-server-v2-swift15]] — the abliterated sibling on the iGPU
- [[frank-archon-radiance-qwen38-iq4r]] — the censored-base flagship this payload shares base alignment with
- [[deployment]] — deploy flow and incidents
serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai9-mtp.md
-362
View File
@@ -1,362 +0,0 @@
---
type: reference
title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10777), Bonsai-2-9B PQ2_0 with the MTP head, 2 × 262144, exact f16 KV (ACTIVE)"
description: "The ACTIVE frank-zealot runbook (since 2026-10-18) — the PrismML fork build 10777 of llama.cpp serving prism-ml/Ternary-Bonsai-2-9B-gguf at PQ2_0 on the RTX 3090: TWO conversations of 262,144 tokens (every slot at the model's trained window) over an UNQUANTIZED f16 K + V cache, vision through the publisher's Q8_0 Qwen3-VL projector, and the boot-arm census that put speculative decoding OFF by measurement even though the checkpoint carries the MTP head."
tags:
- llm
- frank-zealot
- dankclaw
- llama-cpp
- gguf
- rtx3090
- mtp
- spec-decode
timestamp: 2026-10-18
---
# frank-zealot — llama.cpp (PrismML) Bonsai-2-9B PQ2_0 + MTP head, 2 × 262,144, exact f16 KV
Live since 2026-10-18, replacing the
[[frank-zealot-llama-prism-bonsai2-pq20]] era (Ternary-Bonsai-2-**27B** PQ2_0 at
2 × 200,704). The original trade: the 9B bills **half** the KV per token of the
27B, and that is what funded **three** conversations at the model's **native
262,144** window where the 27B afforded only two shortened ones (Frank's ask:
*"if you can possibly fit a third context, do it"*).
The shape moved again the same evening (Frank: *"shift frank-zealot to 2
parallels of unquantized kv"*). The KV cache went **exact** — f16 K + f16 V, no
quantization error anywhere in the attention state — and the **slot count paid
for it: 3 → 2**. The per-slot contract never moved; every conversation is still
262,144, still the trained window. What the dtype bought is fidelity, and what
it costs is one concurrent conversation plus the bytes attention reads per
generated token.
## Payload identity
| Item | Value |
|---|---|
| Runtime | `/home/dank/GitPackages/llama-prism-b10777/llama-server` — PrismML **fork** build, `--version` → `0.2.0-dev (build 10777, commit e3de71522)` |
| Checkpoint | `/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf` — 2,675,410,720 B, Hub-LFS sha256 `c79fcdfaf6b7bf47089b50dadcfd11f9ce7d3e96adf3bbefde7912e5ef69e22b` |
| Vision | `/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf` — 624,229,664 B, Hub-LFS sha256 `e0cb6f6988cea323ef798cd77d019bbb819af75582f034eaab0e7232314953db` |
| Fetch | `scripts/download-frank-zealot-bonsai9.sh [DEST]` (`--verify-only`; `-f`, byte-count and LFS sha256 per file) |
| Alias | `frank-zealot` (`ALIAS`); the SERVICE/unit/scripts keep the StarCraft name, only the wire id is the model id |
| Port | 7771, root systemd unit `frank-zealot.service` (relinked), `TimeoutStopSec=180` |
### The runtime lock has two tumblers now
1. **The tensor type.** `general.file_type 141` with **202 tensors of the fork's
PQ2_0 type** (of 442 total; the rest are f32 norms/scalars and 8 q8_0 in the
MTP block) plus `prism.hadamard.*` metadata
(`normalized-sylvester-walsh-hadamard`, `block_size 4096`,
`axis input-last-dimension`, `sign_mode explicit` over 201 weight names,
inverse on `token_embd.weight`, `gdn_v_grouped true`). Stock llama.cpp
refuses the custom type at load; a loader that ignored the Hadamard metadata
would dequant **wrong**, silently. Same lock class as the ROCmFPX era.
2. **The build floor.** The fork's card gates **MTP beside a quantized V cache**
on `prism-b10776-d00e252` or newer, which is why the prebuilt moved
**10754 → 10777** for the swap. Exact f16 KV makes that gate moot — there is
no quantized V to pair with the drafter — but tumbler 1 pins the runtime on
its own: stock llama.cpp cannot load this file at any flag value. The 10754
tree stays on disk (`~/GitPackages/llama-prism/`) as the 27B rollback
runtime. `tests/frank_zealot_unit.bats` pins both paths.
## Header census (read from the GGUF, not the card)
`general.architecture qwen35`, `general.name Ternary-Bonsai-2-9B`,
`general.size_label 9B`, `general.quantization_version 2`,
`general.file_type 141`:
| Key | Value | Note |
|---|---|---|
| `qwen35.block_count` | **33** | 32 trunk blocks + `blk.32` = the MTP block |
| `qwen35.full_attention_interval` | 4 | ⇒ **8 full-attention + 24 Gated DeltaNet** trunk layers |
| `qwen35.context_length` | 262,144 | native; the deployed slot equals it |
| `qwen35.embedding_length` / `feed_forward_length` | 4,096 / 12,288 | |
| `attention.head_count` / `head_count_kv` | 16 / **4** | |
| `attention.key_length` / `value_length` | 256 / 256 | |
| `ssm.conv_kernel` / `state_size` / `group_count` | 4 / 128 / 16 | the 27B: 4 / 128 / 16 |
| `ssm.time_step_rank` / `inner_size` | **32 / 4,096** | the 27B: 48 / 6,144 |
| `rope.freq_base` / `dimension_count` / `dimension_sections` | 1e7 / 64 / `[11,11,10,0]` | |
| `qwen35.nextn_predict_layers` | **1** | the MTP head is IN the file |
| `general.sampling.*` | temp 1.0 / top_p 0.95 / top_k 20 / min_p 0.0 / repeat 1.0 | the card recipe; llama.cpp reads it itself |
**KV accounting.** Only the 8 full-attention layers scale with context:
8 × 4 KV heads × 256 dim = **8,192 elements/token per cache** (the engine prints
`n_embd_k_gqa = n_embd_v_gqa = 1024`). **The deployed pair is exact: f16 K +
f16 V bills 32,768 B/token** (2 B/element × 8,192 elements × 2 caches), so the
524,288-position pool is **16,384 MiB**. The q8_0 pair this endpoint shipped
first billed **17,408 B/token** — 1.0625 B/element, 32 codes plus one 2-byte
scale per 32-block — the **13,056.00 MiB** its ledger printed at the 786,432
pool, exactly HALF the 27B's 34,816 B/token.
**Corrected arithmetic, and it matters.** The note that lived here and in
`frank-zealot.env` — "f16 would bill 26,112 MiB" — doubled the q8_0 figure
*including* its scale overhead, overstating exact KV by 6.25 %. Exact f16 at the
786,432 pool is **24,576 MiB** against the **24,123 MiB** the driver leaves
addressable (`nvidia-smi`'s total column reads 24,576; used + free reads 24,123
— the difference is driver reservation). The wall stands in both readings; the
honest rate is what makes the fit calculable instead of superstitious. **Halving
the POOL is what funds exact KV** — the window is already at the native ceiling
and there is no YaRN in this path.
The 24 DeltaNet layers bill a fixed recurrent state per SLOT instead: the 3-slot
ledger printed `RS buffer size = 150.75 MiB (3 cells, 32 layers, 3 seqs)` ≈
**50.25 MiB a cell**, so two slots bill 100.50. The SPEC-ARMED arms printed
452.25 MiB — MTP depth 2 bills three seqs per slot, so the drafter triples even
the recurrent state. Pool arithmetic alone under-prices the shape.
## The shape (`scripts/frank-zealot.env`)
`--ctx-size 524288 --parallel 2` (⇒ `n_ctx_slot = 262,144`), `--flash-attn on`,
`--cache-type-k f16 --cache-type-v f16`, `--batch-size 4096`,
`--ubatch-size 2048`, `--n-gpu-layers 99`, `--jinja`, `--reasoning auto`,
`--reasoning-format deepseek`, `--metrics`, `--mmproj …`, `--log-verbosity 4`,
`--presence-penalty 1.0`,
**`SPEC_TYPE=none`** (which strips `--spec-type` AND every `--spec-draft-*`).
Per-slot window is the **contract** the harness quotes: `models.yml` carries
`contextWindow: 262144`, and it did **not** move with the slot count because
llama.cpp divides `--ctx-size` across slots. It refuses a prompt longer than the
SLOT with a 400 naming the limit — never a truncation — and since this swap the
slot window equals the native window, so no YaRN and no cap is in play.
`tests/frank_zealot_unit.bats` now gates the arithmetic: the shipped dtype ×
pool plus the measured non-KV ledger terms must fit 24,123 MiB with ≥1 GiB of
slack, and the gate is proven to REFUSE the impossible arm (exact KV at the old
786,432 pool = 24,576 MiB of cache alone).
## The MTP verdict: the head is in the file and it is still OFF
The drafter is **good**: acceptance 0.798 / 0.617 per position, mean accepted
length 2.41, and the card recommends depth 2. It is OFF because the draft
context bills its own **f16** KV over the WHOLE pool — `blk.32` carries real
attention (4 KV heads × 256 × 2 caches × 2 B = **4,096 B/position** at the
default dtype). The draft dtype is itself a knob — `--spec-draft-type-k` /
`--spec-draft-type-v`, both default f16, verified against this build's `--help`
(so the "no flag exists" claim that lived here is FALSE) — and quantizing the
draft to q8_0 would bill 2,176 B/position, **1,088 MiB** at the shipped pool.
**No measured arm runs a drafter beside exact KV**; every row below is f16 draft.
The census below is the **q8_0 era**: every row is q8_0 K + q8_0 V at the
786,432 pool, where the draft bill was 3,072.00 MiB and 4,830 MiB with its
compute buffers. Booted arms, same card, same traffic, 2026-10-18
(`/completion` `timings`, greedy 512 tokens; never the first request after a
boot):
| slots | spec | ubatch | used / free | decode | 3-stream | prefill @106k |
|---|---|---|---|---|---|---|
| **3 × 262,144** | **off** | **2048** | **17,876 / 6,249** | **167.2** | **280.2** | **2,877** |
| 3 × 262,144 | on | 2048 | 22,706 / 1,419 | 109.2 | — | — |
| 3 × 262,144 | on | 1024 | 21,400 / 2,725 | 213.5 | 250.8 | 2,395 |
| 2 × 262,144 | on | 2048 | 17,178 / 6,947 | 213.2 | — | — |
| 1 × 262,144 | on | 2048 | 11,652 / 12,473 | 214.2 | — | — |
**MTP is worth +28 % single-stream decode whenever the card has room** (rows
4–5 prove the drafter itself is a win; row 2 proves the squeeze is what halves
it). The arm that ran it at three windows was `SPEC_TYPE=draft-mtp` +
`UBATCH_SIZE=1024` (row 3), paying **prefill −17 %** and **2,725 MiB of
headroom** on a card whose mmproj worst-case lands AFTER the engine's own fit
check.
**Under exact KV an f16-draft arm is priced out of this card.** The draft
context is built at the TARGET's pool, so at 524,288 positions it bills
4,096 × 524,288 = **2,048.00 MiB** plus its own buffers, on a trunk that already
projects **~21,200 MiB** of the 24,123 MiB addressable. The doors back to
speculation are **one window per conversation** (row 5's class of shape), **KV
back to q8_0** with `UBATCH_SIZE=1024`, or the **unmeasured**
`--spec-draft-type-k q8_0 --spec-draft-type-v q8_0` arm — 1,088 MiB of draft,
legal on b10777, never booted beside exact KV. Frank chose exact KV over the
drafter on 2026-10-18, and the endpoint's actual job is what made that
affordable: the fleet's **compaction hub** is prefill-bound on ~170k inputs,
prefill is weight-bandwidth bound and does not re-read the cache per generated
token — so the dtype is paid in decode and concurrency, not in the work this
service exists to do.
## Boot ledger — the q8_0 3-slot production boot (2026-10-18 13:19, under the unit, verbosity 4)
```
CUDA0 model buffer size = 2036.71 MiB # + 257.66 MiB CPU_Mapped (mmap)
CUDA0 KV buffer size = 13056.00 MiB # q8_0 K + q8_0 V, 786,432 positions
CUDA0 RS buffer size = 150.75 MiB # 3 cells, 32 layers, 3 seqs
CUDA0 compute buffer = 1409.02 MiB # + 1088.87 MiB CUDA_Host, 2.84 output
mmproj worst-case (est.) = 868.32 MiB # 248.10 CUDA + 24.93 CPU, AFTER fit
nvidia-smi settled: 17,922 MiB used / 6,203 MiB free of 24,123
n_slots = 3, n_ctx_slot = 262144, kv_unified = false
```
No clamp line, no `cudaMalloc` error. The engine's own projection (21,439-class
numbers from the 27B era) excludes the projector — **read settled `nvidia-smi`,
never the projection**.
**The exact-KV arm's projection, recorded before its own boot.** Every term of
that ledger survives the reslice except the KV pool (13,056 → 16,384 MiB) and
one recurrent-state cell (−50.25 MiB), so the pre-flight prediction is
`17,922 − 13,056 + 16,384 − 50 ≈ 21,200 MiB used / ~2,920 free`, with ~2 GiB
still standing through the mmproj worst case. **Replace this paragraph with the
MEASURED ledger and settled `nvidia-smi` from the first f16 boot** — a
projection is a pre-flight check, not a record, and the whole reason the shape
changed dtype is that projections drift from ledgers.
## Measured — the q8_0 3-slot boot (`/completion` `timings`, greedy, `ignore_eos`)
- **168.7 t/s** decode — three runs of 512 tokens: 168.4 / 168.7 / 168.7
(the trial arm printed 167.2; same shape, same noise band).
- **~300 t/s aggregate** across three concurrent 512-token streams
(102.8 t/s per stream, round 1 and 2 both) — the trial arm's 280.2.
- **2,877 t/s prefill at 106,418 tokens** — the trial arm; the production boot
has not re-run that depth.
- The 27B record it replaces: 76.6 decode, 109.4 aggregate across two streams,
691.7 prefill at 166k. Beaten on every axis.
- The **first request after a boot is a warmup artifact** (37.7 t/s class) —
never report it. And never time a wall-clock aggregate on a generation that
stops on its own: a 24-token countdown measured 72 t/s across three streams
while the same shape with `ignore_eos` gave 300.
**These are the numbers the exact-KV arm is judged against.** Exact KV makes
attention read 1.882× the bytes per generated token, so a decode drop is
EXPECTED and has to be measured, not assumed; prefill should hold. Re-run the
same instrument on the first f16 boot — greedy, `ignore_eos`, 512 tokens, the
second request or later, and the two-stream aggregate in place of the
three-stream one — and record it here.
## Contract traps
- **`--metrics` is mandatory.** Without it llama.cpp answers **501** on
`/metrics`, so the Prometheus scrape reads DOWN while `/health` answers 200 —
that is how this endpoint sat red across four payload generations
(`up{job="frank-zealot"}` oldest retained sample, 2026-10-11, is already 0).
- **Q8_0 mmproj, not BF16.** The publisher's card: the BF16 tower "can crash
llama-server when several image requests run at once; Q8_0 is not affected" —
and `--parallel 2` is still precisely that shape.
- **Only the `image_url` content part reaches the model.** The legacy top-level
`images` field bumps `usage.prompt_tokens` and the model still invents an
answer. Probe with a solid color and expect the exact word.
- **≥256 px probe images** or the model answers "video".
- **Thinking tokens count against `max_tokens`.** A vision probe at 64 tokens
returns EMPTY `content` + `finish_reason: "length"` — the endpoint looks blind
while it is only mid-thought. Budget ≥512.
- **`presence_penalty` has no GGUF metadata key**, so the one recipe value that
must reach the argv is this one. The serve script uses
`${PRESENCE_PENALTY=1.0}` — **not `:=`** — because `""` is the documented
opt-out and `:=` would silently re-add the flag (caught by the seam, fixed
2026-10-18).
- **Spec flags move as a SET.** A bare `--model-draft`/`--spec-type` arm without
the whole set has segfaulted at load on this fork family in the past;
`SPEC_TYPE=none` strips all of them.
- **Base alignment is the censored base lineage, and the CRACK swap is NOT
one line here.** `/home/dank/models/Bonsai2-CRACK/` matches the **27B**
geometry, not this 9B file; a dealigned 9B would have to be fetched.
- **A raw `/completion` request** logs `failed to parse reasoning content
(… parse_reasoning=true)` when the model closes a bare prompt with `` —
cosmetic (`probe_request.cpp:789`); `/v1/chat/completions` parses
`reasoning_content` correctly. Do not lower `VERBOSITY`: the KV/buffer ledger
is what VRAM forensics read.
- **A failed stop leaves a `failed` unit, and an orphan keeps answering.** On
2026-10-18 the trial stop hit `State 'stop-sigterm' timed out. Killing.` →
`Active: failed (Result: timeout)`, while the hand-booted arm kept serving
7771 — a live port with no unit behind it. `TimeoutStopSec=180` is now on the
unit (llama.cpp needs the drain time to release slots and the card); the
hand-booted arm must be killed by PID, never `pkill -f llama-server`.
- **A leftover arm drop-in hijacks the production boot, and no health check
sees it.** The sanctioned bench method is a transient
`Environment=FRANK_ZEALOT_ENV_FILE=/tmp/<arm>.env` drop-in, and an arm env
`source`s the repo env and then overrides it. On 2026-10-18 a stale
`frank-zealot.service.d/ptq1.conf` — written 03:29 for the 27B **PTQ1_0** arm
Frank had ordered at 02:36 and measured from 03:14 (deploy ledger below) —
was still installed, so the cutover deploy restarted
the unit and booted THAT: `--model …/Bonsai2-PTQ1_0/Ternary-Bonsai-2-27B-PTQ1_0.gguf
--ctx-size 401408 --parallel 3 --batch-size 3072 --ubatch-size 1536`, three
slots of 133,888, 21,188 MiB used. `/health` was 200, `/v1/models` said
`frank-zealot` (the alias is a flag, not a fact), and the Playwright gate
passed 33/33 — none of those read the payload. **After deploying an endpoint
that was recently benched: `systemctl show -p DropInPaths`, then
`tr '\0' ' ' </proc/$pid/cmdline`, then the ledger — in that order.** Retire
the drop-in to `~/.AGENTTRASH/<date>-<topic>/`, `daemon-reload`, restart.
## Compaction hub
`models.yml` routes `compactionModel` for `frank-archon`, `frank-dragoon` and
`frank-zealot` here — 262,144 is the widest window in the fleet and the only one
that comfortably fits a worst-case compaction input (~170k tokens; the clamp is
`min(maxTokens 16384, contextWindow − est − 4096)`, compaction fires at 85 % of
the window ≈ 222,822 est, worst case ≈ 245,891 ≤ 262,144). What the reslice to
**two** slots does cost is here: a compaction request still runs beside ONE live
generation, but with two conversations already busy a third waits — the Gemma4
era's `--parallel 1` serialization law at half strength. The window itself is
untouched, and the window is the term the compaction contract depends on.
## Deploy ledger (2026-10-18)
- `./scripts/deploy.sh dankclaw --service frank-zealot` at 13:16 PDT; the e2e
gate ran 33/33. The first boot was the hijacked arm above; the drop-in went
to `~/.AGENTTRASH/20261018-zealot-ptq1-arm/` and the 13:19 restart is the
ledger recorded here.
- Live probes on the shipped boot: chat `27*59` → **1593**; `where_am_i` tool
call fired with schema-correct empty arguments; 512 px solid-red PNG →
**Red** at 278 prompt tokens with `process_mtmd` chunk encoding in the
journal; `/props` reports `n_ctx 262144` and `presence_penalty 1.0`;
`/metrics` answers 200 with 15 `llamacpp:*` families and
`up{job="frank-zealot"}` is **1** in Prometheus for the first time in its
retained history.
- OpenWebUI picker rows re-keyed in the live DB (`Connection.backup()` snapshot
`webui.db.bak-20261018` taken first; `frank-dragoon`'s `context.length` also
corrected 262,144 → 200,704 — see [[OpenWebUI]]).
- **What the bench window left behind is not garbage — the PTQ1_0 arm was
ordered, measured, and then scratched by Frank himself.** The instruction at
**02:36** was: run frank-zealot on `Ternary-Bonsai-2-27B-PTQ1_0.gguf`, the
**Q8** mmproj, batch/ubatch **3072/1536**, and as much context as fits in
**two** contexts. The unit booted it at **03:14** (journal:
`type ptq1_0: 402 tensors`, `file type = PTQ1_0 - 1.75 bpw ternary`, mmproj
loaded, 2 × 200,704) and served 24 minutes of traffic. Its own
`print_timing` lines measure **~48 t/s** single-stream decode
(48.0 / 47.9 / 47.6 / 49.0 t/s on 512 / 229 / 512 / 451-token generations),
25–27 t/s per stream with two streams live, and prefill **1,295 t/s at
30,637 → 943 at 97,856 → 801 t/s at 141,365** tokens. It bought the
1,201 MiB the fetch script promised and paid for it on decode, exactly as the
publisher's Ampere row warned (the PQ2_0 record for the same weights is
76.6 t/s). At **03:33** Frank scratched the whole 27B plan and ordered the
9B PQ2_0 + MTP shape deployed here; the arm process ended at 03:38.
- The actual failure was **documentation, not measurement.** An approved
payload that survives only as a systemd drop-in plus a journal window is,
a day later, indistinguishable from a rogue arm — which is why the 13:16 boot
read as a hijack. Payload decisions belong in this runbook the moment they
are made. The evidence is still perishable: `/tmp/zealot-arm-current.env` and
the 9B `/tmp/arm-*.jsonl` results die on reboot, and only the journal
(`journalctl -u frank-zealot --since "2026-10-18 03:10"`) holds the PTQ1_0
numbers. `/home/dank/models/Bonsai2-PTQ1_0/` (5,946,648,928 B + hardlinked
mmproj) stays on disk as the 27B rollback.
## Fallback rungs
Cheapest first; every one is a single line of `scripts/frank-zealot.env`, and
the KV figures are for the 524,288-position pool unless stated otherwise:
1. `UBATCH_SIZE=1024` — hands back **1,306 MiB** of compute buffers measured on
this card (prefill −17 %).
2. `CACHE_TYPE_V=q8_0` — exact K, quantized V: the pool goes 16,384 →
**12,544 MiB**, freeing 3,840, and it is the halfway door back.
`--flash-attn on` is what keeps quantized V legal.
3. `CTX_SIZE=401408` — two 200,704 conversations: exact KV at
**12,544.00 MiB**, same 3,840 freed, but `models.yml` `contextWindow` MUST
follow the per-slot number or long turns start 400-ing.
4. Revert to `q8_0` K + `q8_0` V at `CTX_SIZE=786432 PARALLEL=3` — the shape
this one replaced: 13,056.00 MiB of cache and the THIRD conversation back.
5. `CACHE_TYPE_K/V=q4_0` — the far rung: 10,240 B/token ⇒ **5,120 MiB** here
(the 27B era measured q4_0 at 16,392 used / 7,733 free at 2 × 200,704).
Any retune has to keep the seam's fit gate green — it is the only instrument
that can see a boot that would die at `cudaMalloc`.
## Rollback
Revert the swap commit. On disk: the **27B payload**
(`/home/dank/models/Bonsai2-PQ2_0/`) + the **10754** fork tree, the Qwen3.5-9B
Q6_K payload (`/home/dank/models/Qwen35-9B-HauhauCS-Q6K/`) + master CUDA build,
the Gemma4, MiMo, Swift and Ornith payloads. Then
`./scripts/deploy.sh dankclaw --service frank-zealot`.
## Related
- [[frank-zealot-llama-prism-bonsai2-pq20]] — the retired 27B era (rollback target)
- [[Llama Cpp Serving Sizing]] — the two-boot VRAM pricing method
- [[frank-dragoon-flash-server-v2-native]] — the other 2026-10-18 cutover
- [[deployment]] — deploy flow and incidents
serverAppConfigs-docs/LLM/frank-zealot-llama-qwen35-9b-q6k-2x200704.md
-167
View File
@@ -1,167 +0,0 @@
---
type: reference
title: "frank-zealot on Dankclaw — llama.cpp, Qwen3.5-9B Q6_K, 2 x 200704, f16 KV (RETIRED, rollback target)"
description: "The frank-zealot runbook of 2026-10-07 → 2026-10-17 (RETIRED — rollback target) — llama.cpp master (CUDA sm_86) serving HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K on the RTX 3090: two conversations of 200,704 tokens with exact f16 KV, vision through the publisher's BF16 projector, MTP impossible (no tensors in the file), and the measured geometry that priced the shape."
tags:
- llm
- frank-zealot
- dankclaw
- llama-cpp
- gguf
- rtx3090
timestamp: 2026-10-07
---
# frank-zealot — llama.cpp Qwen3.5-9B Q6_K, 2 × 200,704, f16 KV
**RETIRED 2026-10-07 → 2026-10-17** (AGENTS.md Runbook Index), superseded by
[[frank-zealot-llama-prism-bonsai2-pq20]] (27B PQ2_0, 2 × 200,704 q8_0 KV, build
10754), itself retired 2026-10-17 → 10-18 and now the rollback target: :7771
today is llama.cpp PrismML build **10777** on
`Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, 2 × 262,144 exact f16 KV, MTP off by
measurement (AGENTS.md Runbook Index active row; `scripts/frank-zealot.env`).
Transient 2026-10-18: :7771 is inside a bench window — the unit is `failed`
since 03:14 and a hand-booted process answers the port; restore with
`./scripts/deploy.sh dankclaw --service frank-zealot`.
This era is the rollback target: the payload is still on disk
(`stat -c %s /home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf`
→ 7359259008, read 2026-10-18) with its master CUDA build. Same census: the
10754 rollback payload `Bonsai2-PQ2_0/Ternary-Bonsai-2-27B-PQ2_0.gguf` is
present at 7206168928 with its Q8_0 mmproj at 629246976, and
`Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf` at 2675410720 —
nothing this era or its successors need has been purged. It replaced the
[[frank-zealot-llama-gemma4-q4km-200704]] era (Q4_K_M on ONE 200,704
slot). The swap is a capacity trade: the 9B dense-hybrid rung costs ~7 GiB of
weights where the 26B MoE cost ~16.6 GiB, and the freed room buys the second
conversation slot and unquantized KV back.
## Identity
| | |
|---|---|
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
| Endpoint | `http://192.168.0.143:7771` — served model id **`frank-zealot`** (`ALIAS` in the env; the SERVICE keeps the `frank-zealot` name), `/v1` OpenAI surface |
| Runtime | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` — `--version` on the disk build reads `0.5.0-dev (build 1, commit 5fc4f3c)` (built 2026-10-01); its `CMakeCache.txt` says `GGML_CUDA_FA_ALL_QUANTS:BOOL=OFF` and that tree carries no CUDA-side guard for the option (`ggml/src/ggml-cuda/fattn.cu:438-449` dispatches Q4/Q5 KV unconditionally), so quantized KV never required it; `qwen35` is registered in the build's own `src/llama-arch.cpp` and `src/models/qwen35.cpp` exists at that commit |
| Unit | root systemd `frank-zealot.service` ([`scripts/frank-zealot.service`](../../scripts/frank-zealot.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
| Serve script | [`scripts/serve-frank-zealot.sh`](../../scripts/serve-frank-zealot.sh), tunables [`scripts/frank-zealot.env`](../../scripts/frank-zealot.env) |
| Weights | `/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf` — HauhauCS **Q6_K**, 7,359,259,008 B, sha256 `c0ba7beb68fd3fe47891bd549486d38dcf62d00817296ea314ad37017f5a4986` |
| Projector | `mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf`, 921,704,448 B, sha256 `05f662501f8bd45607b079723a3e238a4e888fd085a10a53f4057a0e250f6934` — the model is natively multimodal and the Q6_K file carries no vision tensors, so this file IS the vision capability |
| Fetch | [`scripts/download-frank-zealot.sh`](../../scripts/download-frank-zealot.sh) — `-f`, byte-count + sha256 verified (the hashes are computed locally: the repo is Xet-backed, so the Hub publishes file OIDs, not file sha256), idempotent, `--verify-only` |
| Seam | `tests/frank_zealot_unit.bats` (argv assertions against a stub engine) |
## Payload geometry (read from the GGUF header, not the model card)
Range-read of the file's own metadata + tensor-info section:
|Key|Value|
|---|---|
|`general.architecture`|`qwen35` (dense hybrid: Gated DeltaNet + softmax attention)|
|`qwen35.block_count`|32, `full_attention_interval` 4 ⇒ **8 full-attention layers (3, 7, 11, …, 31) + 24 DeltaNet layers**|
|attention|16 heads, **4 KV heads**, `key_length` = `value_length` = 256|
|`qwen35.ssm.*`|`conv_kernel` 4, `state_size` 128, `group_count` 16, `time_step_rank` 32, `inner_size` 4096|
|context / vocab|262,144 native / 248,320|
|tensor types|Q6_K × 250 + F32 × 177 — standard types only, so the stock master build loads it (the ROCmFPX-style custom-quant lock-in does not apply)|
|MTP|**no `nextn`/`mtp` tensors exist** — in-file MTP is impossible; the card's "MTP support" line describes the base architecture, not this quantized export|
|chat template|supports `enable_thinking`, `tools`/`tool_call`, image/video content parts; it has **no `reasoning_effort` variable** (unused Jinja variables are ignored, so the harness's effort map is inert here, not an error source)|
## Shape
`--ctx-size 401408 --parallel 2` — llama.cpp divides the pool across slots,
so each conversation gets `n_ctx_slot = 200,704` (Frank's cap; the harness
`contextWindow` quotes the per-slot number). `--flash-attn on`,
**`--cache-type-k f16 --cache-type-v f16`** (exact KV), batch 4096 /
**ubatch 2048**, `--log-verbosity 4`, `--jinja --reasoning auto
--reasoning-format deepseek`, `--mmproj`, **speculative decoding OFF**
(`SPEC_TYPE=none` strips both spec flags — the file has no MTP tensors).
**KV rate law.** The pool scales with the 8 full-attention layers: 8 × 4 KV
heads × 256 dim = 8,192 elements/token per cache, so f16 K + f16 V bills
32,768 B/token = **32 KiB/token** → 12.25 GiB at the 401,408 pool. The 24
DeltaNet layers bill a fixed recurrent state **per slot**, not per token —
that is what makes two 200k slots affordable where the 26B MoE fit only one.
**Priced from boots, not formulas.** MiMo-V2.6-Distill-Qwen-9B (identical
qwen35 geometry, 7,793,710,624 B, same mmproj class) at 2 × 200,704 f16 K +
f16 V measured **21,276 used / 2,849 free** at ubatch 1024 and **21,846 /
2,279** at ubatch 2048 — see [[frank-zealot-llama-mimo-q6k-2x200704]].
This file is 415 MiB lighter in weights and its projector is 3 MiB heavier,
so the projection is ~21,430 used / ~2,700 free.
Ubatch 2048 is the measured CUDA optimum for this card and arch (+4.6 %
prefill at 8.9k depth, decode flat, cost 616 MiB); `1024` is the documented
fallback rung if a future shape lands under the 2 GiB floor.
## Measured boot (2026-10-07, deploy `ec6184d`)
`nvidia-smi`: **21,436 MiB used / 2,689 MiB free** — the MiMo-priced
projection (~21,430 / ~2,700) was within 11 MiB. Engine ledger
(`--log-verbosity 4`, `journalctl -u frank-zealot`):
|Line|Value|
|---|---|
|`llama_context`|`n_ctx = 401408`, `n_ctx_seq = 200704`, `kv_unified = false`|
|`llama_kv_cache`|`CUDA0 KV buffer size = 12544.00 MiB` — `size = 12544.00 MiB (200704 cells, 8 layers, 2/2 seqs), K (f16): 6272.00 MiB, V (f16): 6272.00 MiB`|
|`sched_reserve`|`CUDA0 compute buffer size = 1136.06 MiB`, `CUDA_Host compute buffer size = 848.07 MiB`|
|`clip_model_loader` (mmproj)|`model size: 878.99 MiB` + `reserve_compute_meta: CUDA0 248.10 MiB, CPU 24.93 MiB`|
|`srv load_model`|`n_slots = 2, n_ctx_slot = 200704`|
|`print_info`|`n_ctx_train = 262144`, `n_head_kv = 4`|
The `32 KiB/token` law is confirmed exactly: 401,408 × 32 KiB = 12,544 MiB,
which is what the engine printed. The `n_ctx_seq (200704) < n_ctx_train
(262144)` line is informational (a context **below** the trained window is
safe; the Gemma4 era's warning was the opposite case, a drafter running 53 %
**past** its window). Weights load in ~6 s from page cache; `/health` answers
`{"status":"ok"}` and `/v1/models` serves the alias `frank-zealot`.
**Wire rename (2026-10-09):** `ALIAS` moved `frank-secondary` → `frank-zealot`
(commit `7c77326`; the service-file rename to the `frank-zealot.*` names
followed 2026-10-10, commit `55f87fc`),
so the served id matches the harness model id and the family-facing name.
Consumers re-keyed in the same change: OpenWebUI (`model` row, `access_grant`
row, `openai.api_configs` index-2 `model_ids`) and `models.yml` (already
renamed 2026-10-08). llama.cpp does NOT validate the requested model name —
it answers and echoes the served alias — so a stale client keeps working;
the picker and any id-keyed grant is what breaks. See [[OpenWebUI]].
The deploy's Playwright gate passed (33 tests) and the fleet e2e set is
unchanged.
## Client contract
llama.cpp surface: model id `frank-zealot`, native tools via `--jinja`,
thinking in **`reasoning_content`**, vision via OpenAI `image_url` content
parts (probe with a ≥ 256 px image, or the Qwen-VL tower answers "video").
The publisher's recipe ships in `models.yml` `extraBody`: thinking mode
`temperature 0.6 / top_p 0.95 / top_k 20 / min_p 0` (non-thinking mode is
0.7/0.8/20/0). The card's advice "maintain at least 128K context to preserve
thinking" is satisfied by the 200,704 slot.
**Compaction hub**: `models.yml` routes `compactionModel` for all three
endpoints here — still true 2026-10-18 (`~/.omp/agent/models.yml:25,107,185` all name `llamacpp (dankclaw-3090)/frank-zealot`). With `--parallel 2` a compaction request no longer serializes
behind a live generation (the Gemma4 era's `--parallel 1` law), and the
200,704 slot still fits the worst-case compaction input (~133k tokens).
## Rollback
|Target|How|
|---|---|
|Gemma4-26B-A4B Q4_K_M, 1 × 200,704, f16 K + q8_0 V|revert the swap commit — payload stays at `/home/dank/models/Gemma4-26B-A4B/`; see [[frank-zealot-llama-gemma4-q4km-200704]]|
|MiMo-V2.6 Q6_K, 2 × 200,704, f16 KV|`MODEL_FILE=/home/dank/models/MiMo-Q6K/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf` + its mmproj — measured 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048|
|Swift-1.5-27B IQ3_XXS 2 × 114,688 + MTP|payload on disk at `/home/dank/models/Swift15-UncIQ3XXS/`|
## When it misbehaves
Same decision tree as the predecessors: `No CUDA GPUs are available` ⇒
`nvidia-modprobe -u -c=0`; a 400 naming 200,704 ⇒ the conversation outgrew
one slot, not the pool; queue behind two long generations ⇒ the slot ceiling
is two, by contract; a boot that OOMs ⇒ read the verbosity-4 ledger in
`journalctl -u frank-zealot` (KV MiB, compute buffers) before touching
flags — the engine prints the exact figures the shape needs.
**`status=203/EXEC` boot loop**: the rewrite of `serve-frank-zealot.sh`
shipped without its `#!/usr/bin/env bash` line, and systemd failed the
`ExecStart` before the script ran — 10 s restarts, `Mem peak: 1.9M`, and
`nvidia-smi` flat at 1 MiB used (the GPU is never touched). Journal signature:
`Failed at step EXEC spawning /home/dank/…/serve-frank-zealot.sh`. A
shebang is not decoration on this host: every LLM serve script is a direct
`ExecStart`, so `head -1` is part of the deploy check (commit `ec6184d`).
serverAppConfigs-docs/LLM/frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688.md
-353
View File
@@ -1,353 +0,0 @@
---
type: reference
title: "frank-zealot on Dankclaw — llama.cpp, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 x 114688, draft-mtp, q8_0 KV"
description: "The frank-zealot runbook of 2026-10-05 → 2026-10-07 (RETIRED — rollback target; live from 2026-10-05 evening, renamed from frank-small 2026-10-06, per-slot lowered to 112k and ubatch raised to 1024 the same evening) — llama.cpp master (CUDA sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS (MTP head in the file, drafted) on the RTX 3090: TWO 114,688-token conversations at Frank's 112k cap, q8_0 K + q8_0 V (llama.cpp's FP8 class), vision through the publisher's own BF16 projector, and the full two-slot + drafter arithmetic."
tags:
- llm
- frank-zealot
- dankclaw
- llama-cpp
- gguf
- rtx3090
- qwen35
- mtp
- vision
timestamp: 2026-10-06
---
# frank-zealot — llama.cpp Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2 × 114,688, draft-mtp, q8_0 KV
> **RETIRED 2026-10-06 → 2026-10-07** (AGENTS.md Runbook Index; cut over
> 2026-10-05 ~23:26, reshaped three times on 2026-10-06, renamed from
> frank-small the same night). Superseded by
> [[frank-zealot-llama-qwen35-9b-q6k-2x200704]]; the production shape on :7771
> today is llama.cpp PrismML build **10777** on
> `Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, 2 × 262,144 exact f16 KV, MTP off by
> measurement (AGENTS.md Runbook Index active row); the 27B PQ2_0 / 2 × 200,704
> shape of [[frank-zealot-llama-prism-bonsai2-pq20]] is that era's rollback
> target. Transient 2026-10-18 bench window on :7771 — the unit is `failed`
> since 03:14 and a hand-booted process answers the port; restore with
> `./scripts/deploy.sh dankclaw --service frank-zealot`).
> **Rollback target**: the payload is still on disk —
> `stat -c %s` on 2026-10-18 gives 10442827392 for
> `/home/dank/models/Swift15-UncIQ3XXS/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf`
> and 931145888 for its `mmproj-BF16.gguf`. A 27B replaces
> the 9B on the same 24 GiB card: KV at FP8 class, vision on. The
> morning order (drop MTP, ubatch 512, TWO 128k contexts) bought the
> second slot — 1,397 MiB free. The evening order swapped to the
> UNCELERSED Swift-1.5 at RCO-IQ3_XXS and put MTP back ON — landing
> **489 MiB free**: the MTP draft ctx costs **~2.4 GiB, not the ~0.4
> GiB assumed** (the drafter's KV is built against the target's full
> context, not the 3-token proposal depth), paid for by the lighter
> quant (−1.56 GiB). The late order dropped the per-slot window
> 131,072 → **114,688** (Frank's 112k cap) and climbed the prefill ladder
> ubatch 512 → 1024 → **2048** — landing **1,005 MiB free** (1,751 at
> the 1024 rung; the 2048 rung cost 746 MiB, ~130 over the 616 MiB
> measured prediction). This supersedes BOTH earlier 2026-10-05
> payloads: the live MiMo-V2.6 Q6_K shape
> ([[frank-zealot-llama-mimo-q6k-2x200704]], now retired) and the
> Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but
> never deployed.
## Identity
| | |
|---|---|
| Host / device | Dankclaw (192.168.0.143), RTX 3090, 24,576 MiB |
| Endpoint | `http://192.168.0.143:7771` — alias `frank-secondary` on the wire until the 2026-10-09 rename to `frank-zealot` (commit `7c77326`), `/v1` OpenAI surface |
| Runtime | llama.cpp master (0.5.0-dev, commit 5fc4f3c), CUDA sm_86 build `…/llama-master/build-cuda/bin/llama-server` (quantized V needs `--flash-attn on`; it does NOT need `GGML_CUDA_FA_ALL_QUANTS` — see the KV section) |
| Unit | root systemd `frank-zealot.service` ([`scripts/frank-zealot.service`](../../scripts/frank-zealot.service)), `User=dank`, `Restart=on-failure`, `LimitMEMLOCK=infinity` |
| Serve script | [`scripts/serve-frank-zealot.sh`](../../scripts/serve-frank-zealot.sh), tunables [`scripts/frank-zealot.env`](../../scripts/frank-zealot.env) |
| Weights | `/home/dank/models/Swift15-UncIQ3XXS/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf` — ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF **RCO-IQ3_XXS -mtp**, 10,442,827,392 B, sha256 `e13d3e74…` (qwen35 hybrid: 64 real layers, 16 full-attention + 48 Gated DeltaNet, native 262,144; **MTP head in the file**: blk.64.nextn.*, `nextn_predict_layers=1`, and DRAFTED with it) |
| Projector | `…/Swift15-UncIQ3XXS/mmproj-BF16.gguf` — PUBLISHER-SHIPPED in the same repo, 931,145,888 B, sha256 `19acd7fc…` (27-layer ViT → 5120) |
| Fetch | [`scripts/download-frank-zealot.sh`](../../scripts/download-frank-zealot.sh) — `-f`, byte count + sha256 verified per file (Xet repo: the Hub APIs expose no LFS oid, so the shas were computed locally after the first verified fetch; byte counts match the Hub's declared sizes) |
| Seam | `tests/frank_zealot_unit.bats` (argv assertions against a stub engine: payload path, slot geometry, KV pair, spec pair (both flags present), batch pair, projector opt-out) |
## Why this payload (2026-10-06 evening)
Frank asked for the UNCELERSED Swift-1.5 with MTP back on. The RCO-IQ3_XXS
is 1.56 GiB lighter than the IQ3_S-mtp it replaced — and that lighter
weight is exactly what funds the MTP draft context (~2.4 GiB measured)
PLUS the second 131,072 slot, while keeping the q8_0 KV contract and the
two-slot shape Frank ordered that morning.
**Uncensored, intentionally:** the publisher reports 23/100 refusals vs
Swift 1.5's 98/100 (KL 0.0884 against the censored base). The refusal
escape hatch is the same as every payload here: revert and redeploy.
## Shape and KV contract
`--ctx-size 229376 --parallel 2` — TWO conversations at Frank's 112k
cap (2026-10-06 late: cut from the 128k the morning/evening shapes ran
at, to fund ubatch 1024). `CTX_SIZE` stays divisible by `PARALLEL`; the
seam enforces it, and the per-slot number (`229376 / 2 = 114,688`) is
the product contract — models.yml follows at 114,688 (a longer prompt
is a 400, never a truncation).
**KV at FP8 class: `q8_0` K + `q8_0` V (Frank, 2026-10-05 evening).**
llama.cpp has no true fp8 KV dtype; q8_0 is the 8-bit class (1.0625 B/elem
⇒ 8.5 effective bits) and the only FP8-class lever this runtime moves.
The order supersedes the same-day f16/f16 rule, which was written for the
Ornith 9B fit — at 64 KiB/token f16 the single 131,072 slot alone would
cost 8.0 GiB of KV. At q8_0 the rate is **34 KiB/token** (16 attn layers ×
4 KV heads × 256 dim × 1.0625 B, K and again V). `--flash-attn on` is
pinned: quantized V requires it. The era note claiming the build carried
`GGML_CUDA_FA_ALL_QUANTS=ON` does not describe the binary on disk: its
`CMakeCache.txt` reads `GGML_CUDA_FA_ALL_QUANTS:BOOL=OFF` (binary built
2026-10-01 22:35) and the option is vestigial in that tree — `ggml/src/ggml-cuda/fattn.cu:438-449`
dispatches the Q4/Q5 KV cases unconditionally.
**Two slots + drafter, by arithmetic and by order (2026-10-06 morning /
evening, the 2 × 131,072 era).** The morning order bought the second slot
by spending the drafter and ubatch headroom (1,397 MiB free). The evening
swap put the drafter BACK and changed the arithmetic: the draft ctx is
built against the target's FULL context (262,144 — the draft window, not
the 3-token proposal depth), so it costs **~2.4 GiB measured** (23,636 −
22,728 + 1,561), not the ~0.4 GiB the one-slot era assumed. The lighter
quant pays for it: **23,636 used / 489 MiB free** (nvidia-smi `free`
field) — the tightest fit in frank-zealot's history, the 2–3 GiB floor
knowingly violated, and the 512×512 vision transient (34 MiB) still fits
and recovers.
**The late reshape (2 × 114,688, ubatch 2048) is the current fit.** The
per-slot cut frees 2 × 16,384 tokens × 34 KiB ≈ 1.05 GiB of KV; the
ubatch rungs 512 → 1024 → 2048 spend it back on compute buffers (a
prefill knob — decode never moves with it; the rungs cost ~0.3 and
~0.75 GiB measured). Measured boot at 2048: **23,120 used / 1,005 MiB
free**. The drafter's own KV shrank with its target (draft window
229,376). Compute buffers allocate at graph capture during BOOT, so a
miss is loud and fast — the failure mode is a crash loop, not silent
corruption.
## Speculative decoding — ON since 2026-10-06 evening (draft-mtp, depth 3)
**Frank's evening order: MTP back on.** The MTP head ships IN the file —
the `-mtp` GGUF carries `blk.64.nextn.*` (`qwen35.nextn_predict_layers=1`)
— and `--spec-type draft-mtp` loads it against the main model's own
weights (no `--model-draft`, no second draft model — two context-backed
drafters cannot share a process). `SPEC_DRAFT_N_MAX=3` is the proposal
depth. **ngram-mod stays skipped, not swapped in:** on a qwen35 hybrid,
ngram speculation logs *"recurrent state rollback is not compatible with
'ngram-mod' — disabling rollback support"* (fleet lesson from
frank-dragoon's chain measurement, 2026-10-03). A rejected draft on Gated
DeltaNet layers cannot be rewound by KV truncation — without the rollback
checkpoints the DeltaNet state carries phantom transitions and output
degrades silently. The flags move as a set: the serve script emits BOTH
`--spec-type` and `--spec-draft-n-max` together or neither
(`SPEC_TYPE=none` strips both).
## Vision contract
`--mmproj` is the entire vision path (the GGUF carries no vision tensors).
Probes: a solid-color PNG **≥ 256 px** answers its exact color — a tiny
probe undersupplies image tokens and the model invents ("video"); the
legacy top-level `images` field is IGNORED, only the OpenAI `image_url`
content part reaches the model. **New on this payload: the uncensored
model THINKS more than the censored Swift 1.5** — a 64-token budget was
eaten entirely by `reasoning_content` (`finish_reason: length`, empty
`content`); 128 tokens clears it. Token budgets for vision probes must
leave room for thinking.
## Measured: the 1 × 131,072 MTP boot (2026-10-05, historical)
- **Boot ledger**: `creating MTP draft context against the target model
'…/Swift-1.5-Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf'`, `loaded multimodal
model, '…/Swift15-IQ3S/mmproj-Qwen3.8-27B-F16.gguf'`, `n_slots = 1,
n_ctx_slot = 131072` — every expected line, no q8→f16 fallback.
- **VRAM: 19,404 used / 4,721 MiB free** (nvidia-smi `free` field). The
4.5 GiB prediction held; f16 KV would have left ~700 MiB, so this free
number doubles as the q8_0 proof.
- **Decode + MTP acceptance** (draft counts from the live `timings`):
temp 0.8 → 66.1 tok/s, 182/349 accepted (52.1 %); temp 0 → 66.9 tok/s,
166/318; **deployed recipe (temp 1.0/top_p 0.95/top_k 20) → 58.9 tok/s,
159/355 (44.8 %)** — temperature buys fluency, costs acceptance and
rate. For scale: MiMo Q6_K measured 94.4 tok/s on the same card — this
27B at IQ3_S runs ~2/3 the rate with ~3× the capability class.
- **Vision**: 512×512 solid-red PNG → `Red`, `finish_reason: stop`.
- **Template gate**: `reasoning_effort: xhigh` → 200 with
`reasoning_content` populated; `high` → HTTP 500 (Jinja raise) — the
models.yml fold is mandatory, exactly as the MiMo eras found.
- Streaming completion: 200. No first-request silence window (the
halogen-era 45–60 s stall has no analogue here).
## Measured: the 2 × 131,072 no-spec boot (2026-10-06 morning, historical)
- **Boot ledger**: `loaded multimodal model, '…/Swift15-IQ3S/
mmproj-Qwen3.8-27B-F16.gguf'` and NO draft-context line (speculation
off). `/slots`: **2 slots, n_ctx 131,072 each** (pool 262,144).
- **VRAM: 22,728 used / 1,397 MiB free** (nvidia-smi `free` field) — the
predicted 0.3–1.3 GiB window held; the old 2-3 GiB floor is knowingly
violated by Frank's order. After a 512×512 image probe: 1,363 MiB —
the vision transients fit.
- **Decode without the drafter**: 48.0 tok/s (temp 0) / 47.8 (deployed
recipe temp 1.0). `draft_n` absent from `timings` (no drafter). The MTP
era's 58.9–66.9 tok/s is the honest price of the second context:
−18–28 %.
- **Vision**: 512×512 solid-red PNG → `Red` again on this shape.
- Deploy: commit 812d9eb, e2e gate 33/33 green on the live fleet.
## Measured: the 2 × 131,072 draft-mtp boot (2026-10-06 evening, historical)
- **Boot ledger**: `creating MTP draft context against the target model
'…/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf'`,
`loaded multimodal model, '…/Swift15-UncIQ3XXS/mmproj-BF16.gguf'`,
`n_slots = 2, n_ctx_slot = 131072` — every expected line, no q8→f16
fallback.
- **VRAM: 23,636 used / 489 MiB free** (nvidia-smi `free` field) — the
draft ctx measured ~2.4 GiB, ~6× the ~0.4 GiB the one-slot era
assumed; the lighter quant (−1.56 GiB) plus the morning's no-spec base
still fund both slots. After a 512×512 image probe: 489 MiB — the vision
transient (34 MiB) fits and recovers.
- **Decode + MTP acceptance** (draft counts from the live `timings`,
512-token essay): temp 0 → **72.9 tok/s**, 333/534 accepted (62.4 %);
deployed recipe (temp 1.0/top_p 0.95/top_k 20) → **63.3 tok/s**,
305/612 (49.8 %). e2e traffic in the boot window read 0.52–0.67
acceptance. Against the no-spec shape (48.0 tok/s): **+31 % at the
recipe temp** — the drafter earned its 2.4 GiB.
- **Vision**: 512×512 solid-red PNG → `Red` at a 128-token budget (64
tokens: all thinking, empty content). `prompt_tokens: 321` confirms the
image part reached the model.
- **Prompt cache**: `cached_tokens: 317` of 321 on the second probe — the
slot prefix cache is live.
- Deploy: commit da36fa3, e2e gate 33/33 green on the live fleet.
## Measured: the 2 × 114,688 / ubatch-1024 boot (2026-10-06 late, historical)
- **argv** (from `/proc/<MainPID>/cmdline`): `--ctx-size 229376
--parallel 2` (per-slot 114,688), `--batch-size 4096 --ubatch-size
1024`, `--spec-type draft-mtp`. Deploys 87721ab (the rename — old
`frank-small.service` stopped + disabled first, by hand: `disable` on
a relinked unit removes only its symlink) then f2575a1 (the shape);
health gate + MainPID check green, e2e 32 passed / 1 flaky-retry.
- **VRAM: 22,374 used / 1,751 MiB free** (nvidia-smi `free` field) —
the per-slot cut (≈1.05 GiB of KV) paid for the ubatch-1024 buffers
and netted back ~1.25 GiB against the evening's 489 MiB; back above
the morning's 1,397 MiB.
- **Decode + MTP acceptance** (live `timings`): recipe (temp 1.0) →
**58.2 t/s** (430/1005, 42.8 %); temp 0 → **66.9 t/s** (475/873,
54.4 %). The 2 × 131,072 boot measured 63.3 / 72.9 — −8–10 %, but
acceptance is text-dependent and these prompts differ from that
measurement's, so the comparison is CONFOUNDED, not a shape verdict;
a clean A/B needs stop + mask (bench policy).
- **Smoke**: `READY` completion + `reasoning_content` populated; the
harness reached :7771 under the new alias (models.yml provider/id
`frank-zealot`, config.yml modelRoles rewritten; per-slot
`contextWindow: 114688`).
## Measured: ubatch 2048 (2026-10-06 midnight, live)
- **argv**: `--ctx-size 229376 --parallel 2 --batch-size 4096
--ubatch-size 2048 --spec-type draft-mtp` (commit d8da7a9, Frank's
"set ubatch to 2k"; deploy green, e2e gate 33/33).
- **VRAM: 23,120 used / 1,005 MiB free at boot** — the 1024 → 2048 rung
cost **746 MiB**, ~130 over the 616 MiB the rung measured on
2026-10-05 (rung cost is shape-dependent, not a constant). The
512×512 solid-red vision probe (`Red`) then took ~240 MiB **and kept
it**: settled **763 MiB free** — on this boot the vision encoder's
buffers stay resident after the request (the evening boot saw its
34 MiB transient return).
- **Decode (ubatch-invariant knob, same prompt as the 1024 rung)**:
recipe → 52.0 t/s (388/1133, 34.2 %) vs 58.2 t/s / 42.8 % at 1024 —
acceptance variance at temp 1.0, not a ubatch effect.
- **Prefill (THE ubatch knob)**: **1,209 t/s @ 3,760 tokens** — first
recorded number for this shape; future rungs compare against this
(no same-prompt 1024 baseline exists; the law predicts ≈ +4.6 %
1024 → 2048).
- **Headroom left ~0.7–1.0 GiB.** ubatch 4096 (= the logical batch, no
micro-splitting) doubles the buffer growth again — not affordable;
**2048 is the ceiling for this shape.**
## Client contract
- Reasoning in `reasoning_content` (`--reasoning-format deepseek`);
thinking tokens COUNT against `max_tokens`. **This payload thinks more
than the censored Swift 1.5** (uncensored finetune) — short `max_tokens`
budgets can be eaten entirely by thinking (`finish_reason: length`,
empty content); the 32,768 `maxTokens` in models.yml is comfortably
above.
- The embedded chat template gates `reasoning_effort` to
`{xhigh, medium, low}` — anything else is a raise/HTTP 500, so
`models.yml` `compat.reasoningEffortMap` folds the harness levels onto
that set (mandatory, same as every Qwen3.8-template endpoint).
- Sampling rides the wire (`extraBody`): the publisher recipe temp 1.0 /
top_p 0.95 / top_k 20 / min_p 0 (evaluated at ctx 131,072).
- The RCO-IQ3_XXS tier is a **per-tensor allocation mix** (IQ3_S 97 /
BF16 96 / IQ3_XXS 83 / IQ2_S 72 / IQ4_XS 33 / IQ2_XS 32 / IQ2_XXS 30 /
Q2_K 28 / Q4_K 19 / Q6_K 8 / IQ1_S 4 / IQ1_M 4) — all stock quant types;
stock llama.cpp loads it. Publisher KLD vs Swift BF16: mean 0.002631,
90th 0.042136, max 34.49 — RCO is the publisher's "more experimental"
family.
## Rollback
**Predecessor shape (same 2 × 114,688, ubatch 1024) is ONE revert
away** — revert d8da7a9 and redeploy. The 2 × 131,072 / ubatch-512
era is one revert further (f2575a1). The 2 × 131,072 no-spec
IQ3_S-mtp shape is one revert further — revert da36fa3 next: the env
trio then ships `SPEC_TYPE=none`, `CTX_SIZE=262144`, `PARALLEL=2`,
`UBATCH_SIZE=512` against `/home/dank/models/Swift15-IQ3S/` (the
IQ3_S-mtp payload, still on disk, measured 48.0 tok/s no-spec). The
1 × 131,072 MTP shape is two reverts further back (812d9eb's parent):
the same IQ3_S-mtp file with `SPEC_TYPE=draft-mtp`, `UBATCH_SIZE=1024`,
`CTX_SIZE=131072`, `PARALLEL=1` (measured 58.9–66.9 tok/s with the head
drafting).
Three payloads on disk, all fetchable by the same script's git-history
recipes:
1. **Swift-1.5 IQ3_S-mtp** — the payload live until this cutover:
`/home/dank/models/Swift15-IQ3S/`; the IQ3_XXS swap above.
2. **MiMo-V2.6 Q6_K 2×200,704 f16 KV** — the payload that was LIVE until
the 2026-10-05 cutover: `/home/dank/models/MiMo-Q6K/`; revert
[[frank-zealot-llama-mimo-q6k-2x200704]] (env trio at commit e5893dc or
380aafa's parent), one `MODEL_FILE` + shape edit.
3. **Ornith-1.5-9B Q8_0** — the committed-but-never-deployed swap:
`/home/dank/models/Ornith-Q8/`; revert to 380aafa's tree (ONE native
262,144 slot, f16/f16 KV).
## Trail
- The Ornith-1.5-9B Q8_0 swap (77decb5 → 380aafa, earlier 2026-10-05) was
committed and pushed but its deploy never completed (blocked at the
whole-tree docs gate); the live service was still MiMo Q6_K when Swift
cut over. Both Ornith payloads were verified on disk before Swift.
- Swift header facts (block_count 65, interval 4, KV geometry, nextn) were
read from the Hub file by HTTP range + struct parse — no download needed
— before the payload was fetched.
- Cutover 2026-10-05 ~23:26: the radiance-cutover session (parallel)
label-committed both sets at Frank's direction — 1919ad7 (frank-archon
radiance) then baf33d0 (this swap: trio, seam, runbooks, fleet docs).
Deploy ran `deploy.sh dankclaw --service frank-zealot`; the service
dispatch and health gate went green; the first e2e run failed ONE spec
(12-crawl, `net::ERR_ABORTED` on `www.franksdank.cloud` — a transient,
unrelated to frank-zealot), green on rerun.
- The unit's `Description=` (MiMo era) was stale at cutover time and was
updated in-repo afterward — the relinked unit picks it up at the next
deploy/restart, not retroactively.
- 2026-10-06 (Frank): drop MTP, skip ngram, ubatch 512, TWO 128k contexts
(812d9eb). The tight fit landed: 1,397 MiB free, vision `Red`, decode
48.0 t/s no-spec. Model of the trade: the drafter's ~0.4 GiB + ubatch
~0.3 GiB bought the second 4,352 MiB KV slot.
- 2026-10-06 (Frank, evening): swap to the UNCELERSED Swift-1.5 at
RCO-IQ3_XXS, MTP back on (da36fa3). The lighter quant (−1.56 GiB) funds
the draft ctx (~2.4 GiB measured — the drafter's KV scales with the
target's full 262,144 context, not the 3-token proposal depth) plus the
second 131,072 slot: 489 MiB free, vision `Red` at a 128-token budget,
decode 63.3 t/s recipe / 72.9 temp 0 (48.0 no-spec).
- 2026-10-06 (late, Frank): the endpoint renamed frank-small →
**frank-secondary** (87721ab — "rename frank-small → frank-secondary", scripts, unit, seam, runbooks,
identifiers; historical references to the OLD FreeToken/CUDA
frank-small, and to the Dankserver service that became frank-medium,
stay frank-small) and the per-slot window cut 131,072 → 114,688 to
fund ubatch 512 → 1024 (f2575a1). Live unit transition done by hand
before the deploy: `systemctl stop frank-small` + `systemctl disable
frank-small`, then `deploy.sh dankclaw --service frank-zealot`.
1,751 MiB free; decode 58.2 t/s recipe / 66.9 temp 0; models.yml +
config.yml retargeted.
- 2026-10-06 (midnight, Frank): "set ubatch to 2k" → `UBATCH_SIZE=2048`
(d8da7a9). Cost 746 MiB: 23,120 used / 1,005 free at boot, settling
**763 MiB free after a vision probe** (encoder buffers stay resident
on this boot). Decode 52.0 t/s recipe on the shared prompt (58.2 at
the 1024 rung — acceptance noise, decode is ubatch-invariant);
prefill 1,209 t/s @ 3,760 tokens (first recorded baseline). 2048 is
the ceiling for this shape; 4096 would not fit.
serverAppConfigs-docs/LLM/frank-zealot-qwen38-gsq-rco.md
-22
View File
@@ -1,22 +0,0 @@
---
type: reference
title: "frank-zealot on Dankclaw — llama.cpp, Qwen3.8-27B GSQ-RCO IQ3_S"
description: "The RETIRED frank-zealot runbook (2026-09-16 → 09-17) — mainline llama.cpp on Dankclaw's RTX 3090 serving ISTA-DASLab Qwen3.8-27B-GSQ-RCO at IQ3_S with the model's own in-file MTP head and the BF16 vision projector: identity, KV arithmetic, deployed flags, the MTP contract, the client contract, and what to check when it misbehaves."
tags:
- llm
- frank-zealot
- dankclaw
- llama.cpp
- rtx3090
- qwen35
- mtp
- vision
timestamp: 2026-09-16
---
**RETIRED 2026-09-16 → 2026-09-17** (AGENTS.md Runbook Index); superseded by
[[frank-zealot-gemma4-12b]], and :7771 today runs
[[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML fork build 10754,
`Ternary-Bonsai-2-27B-PQ2_0.gguf`). The payload is **not** a rollback target any
more: the 2026-10-18 census of `/home/dank/models/` has no GSQ-RCO directory, so
restoring this shape means re-downloading from ISTA-DASLab.
serverAppConfigs-docs/LLM/frank-zealot-sglang-mimo-v26.md
-187
View File
@@ -1,187 +0,0 @@
---
type: reference
title: "frank-zealot on Dankclaw — SGLang, MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (RETIRED)"
description: "The RETIRED frank-zealot runbook (2026-10-01, hours) — SGLang in a self-contained python3.12 venv serving leteschlead MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5 hybrid, 9B dense, in-checkpoint MTP + vision) on the RTX 3090: identity, payload, KV arithmetic, the NEXTN spec-decode ghost, the CUDA 13 nvcc toolchain the JIT needs, and the measured numbers kept for the record. Replaced by [[frank-zealot-llama-mimo-q6k]] the same night."
tags:
- llm
- frank-zealot
- dankclaw
- sglang
- rtx3090
- qwen
- mtp
- vision
- int8
timestamp: 2026-10-01
---
# frank-zealot — SGLang MiMo-V2.6-Distill-Qwen-9B INT8 W8A16 (NEXTN MTP)
> **RETIRED 2026-10-01, hours after cutover** (AGENTS.md Runbook Index). Replaced by
> [[frank-zealot-llama-mimo-q6k]] — llama.cpp CUDA on the MiMo Q6_K GGUF,
> chosen for the asymmetric KV (f16 K + q8_0 V) SGLang cannot express. :7771
> today runs [[frank-zealot-llama-prism-bonsai9-mtp]] (llama.cpp PrismML fork
> build **10777**, `Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf`, 2 × 262,144,
> exact f16 K + f16 V, MTP off by measurement).
> The venv (`/home/dank/GitPackages/sglang-cu128/`) is still on disk; the INT8
> payload was staged to
> `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` on 2026-10-02 and
> **that stage batch is gone** (`ls /home/dank/.AGENTTRASH/`, 2026-10-18 —
> only the 20261005/06/07/10/17 batches remain), so this runtime needs
> `old/download-frank-zealot-sglang-int8.sh` to re-stage.
## Identity
- **Endpoint:** `frank-small` at `192.168.0.143:7771` (Dankclaw) — the id
became `frank-secondary` on 2026-10-06 and `frank-zealot` on 2026-10-09.
- **Runtime:** SGLang (latest) in a self-contained python3.12 venv at
`/home/dank/GitPackages/sglang-cu128/` (torch 2.13.0+cu130,
transformers 5.12.1, flashinfer 0.6.18, sgl-kernel). No container — a root
systemd unit `scripts/frank-zealot.service` (`User=dank`,
`LimitMEMLOCK=infinity`) launched by `scripts/serve-frank-zealot.sh` with
tunables from `scripts/frank-zealot.env`.
- **Model:** `letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound`
(9B dense, 12.0 GiB, INT8 W8A16 AutoRound group-128 symmetric).
## Shape
- **Architecture `qwen3_5` hybrid:** 32 layers = 8 full-attention
(4 KV heads x 256 head_dim, `partial_rotary_factor: 0.25`) + 24 Gated
DeltaNet. `max_position_embeddings: 262144`, and a `qwen3_5_vision`
tower (27 layers) in the safetensors — no separate mmproj file.
- **Context:** two 262,144-token slots at the model's native window
(`--context-length 262144`, `--max-running-requests 2`) against a
**pinned** 606,208-token pool (`--max-total-tokens`) — 2026-10-01
reslice from four 151,552-token slots: 2 x 262,144 = 524,288 leaves
~82k tokens of pool slack, and a compaction request (frank-archon's
trigger 183,501 + a 32,768-token summary budget = 216,269) now fits
one request. A single request may grow to the whole pool — a
170,019-token prompt was verified at the 4-slot shape — the grid is
the two-way concurrency contract. **That compaction arithmetic is
specific to a 262,144-token slot**: the shape that followed on 2026-10-02
ran three slots of 180,224, which 183,501 tokens already exceeds — see *What
three slots cost* in [[frank-zealot-llama-mimo-q5km]]; :7771 today runs two
262,144-token conversations over one 524,288-position pool with exact f16 KV.
- **KV:** `fp8_e4m3` — 16 KiB/token (8 attn layers x 2 x 4 heads x 256
dim x 1 B), pool 9.31 GiB.
- **The memory shape (the 4.5 GiB the naive math misses):** the DeltaNet
ssm state pool is a first-class consumer. The engine autofills the
mamba slot count to spend a fixed ~5.1 GB budget *regardless of dtype*
(fp32 at 110 slots, bf16 at 217 — same 5.1 GB), so both dials are
pinned: `--mamba-ssm-dtype bfloat16 --max-mamba-cache-size 32` = 795
MiB. With that, weights 11.35 GiB + mamba 0.78 GiB + pool 9.31 GiB
leave ~0.8 GB — enough for eager-prefill workspaces, NOT enough for
CUDA-graph capture (autofit's 687,388-token pool leaves 0.53 GB and
dies in capture, or later in the first long request once Triton's
deferred kernel loads eat the remainder). Hence
`--disable-prefill-cuda-graph`; decode graphs are untouched.
- **Quantization:** `--quantization auto-round` — the checkpoint's
`quantization_config.json` declares `quant_method: "auto-round"` and
SGLang loads it natively (vision tower, `embed_tokens` and
`linear_attn.in_proj` stay fp16).
## Spec method
**None — and this is a checkpoint fact, not a tuning choice.** The
`text_config` declares `mtp_num_hidden_layers: 1`, but the shipped
safetensors carry **zero `mtp.*` tensors**: verified 2026-10-01 against
`model.safetensors.index.json` for both this INT8 conversion and the base
`XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B` (32 layers, indices 0–31, no
`mtp.`/`nextn.`/`layers.32` keys in either). With NEXTN enabled, SGLang
boots a 4.24 GiB draft model (`Qwen3_5ForCausalLMMTP`) from nothing and
decodes at `accept len: 1.00, accept rate: 0.00` — every proposal rejected,
paying the 4-token verify tax for no token. The serve script passes no
`--speculative-*` flag, and the seam gates that.
## CUDA 13 toolchain (the sharp edge)
The venv's torch is cu130, so every JIT kernel (sgl-kernel marlin repack,
flashinfer) compiles with a **CUDA 13.0 nvcc extracted from debs into the
venv** at `cuda-nvcc/usr/local/cuda-13.0/` — the system nvcc is 12.4 (no
`--compress-mode`) and its headers clash with the 13.0 device passes.
Components (all from the ubuntu2404 CUDA repo, `dpkg -x`'d, not installed):
| Package | Supplies |
|---|---|
| `cuda-nvcc-13-0` (13.0.88) | `bin/nvcc`, `ptxas`, `cudafe++`, … |
| `libnvvm-13-0` (13.0.48) | `nvvm/bin/cicc` (the device compiler) |
| `cuda-cudart-dev-13-0` (13.0.88) | `include/cuda_runtime.h`, `lib64/stubs` |
| `cuda-crt-13-0` (13.0.88) | `include/crt/host_runtime.h` (2-arg `__cudaLaunch`) |
| `libnvptxcompiler-13-0` (13.0.88) | static lib + `nvPTXCompiler.h` |
| `cuda-cccl-13-0` (13.0.50) | `cccl/cuda/...` headers (libcudacxx) |
Three local modifications the tree carries:
1. **`bin/nvcc` is a shim** over `bin/nvcc.real` adding
`-allow-unsupported-compiler` (host GCC 16 > nvcc's max 13).
2. **`targets/x86_64-linux/include` and `lib` are symlinks** to
`../../../include` / `../../../lib64` (the deb ships them as real
near-empty dirs; the stub pass must see the 13.0 headers, not
`/usr/include` 12.4).
3. **`include/cuda` → `cccl/cuda`** symlink (CCCL 3.x layout; flashinfer
includes `<cuda/functional>`), and **`crt/math_functions.h`** patched with
`noexcept` on `rsqrt`/`rsqrtf` (glibc 2.42 declares them `noexcept(true)`;
the unpatched 13.0 declarations are incompatible in the C++ host stub).
JIT caches: `~/.cache/sglang/jit/sm86/` and
`~/.cache/sglang/.cache/flashinfer/0.6.18/86/` — clear both when a toolchain
change lands.
## Payload
Eight safetensors shards (12,891,718,440 B total) + tokenizer and config
files lived under `/home/dank/models/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound/`
(absent from the 2026-10-18 census — see the banner),
all LFS sha256-verified by `old/download-frank-zealot-sglang-int8.sh`
(`--verify-only` re-checks) — the script moved to `old/` when the endpoint
cut over to llama.cpp + GGUF on 2026-10-01 night, and
`scripts/download-frank-zealot.sh` now fetches the GGUF quant instead. The
serve script refuses to start with a missing shard.
## Client contract
- **Alias:** `frank-small` (the `id` at `/v1/models`).
- **Reasoning:** extracted to **`reasoning_content`** (live-verified
2026-10-01 — this SGLang build's qwen3 parser uses the same field name
as llama.cpp; models.yml sets `reasoningContentField:
"reasoning_content"`). `--reasoning-parser qwen3`.
- **Tools:** `--tool-call-parser qwen3_coder`.
- **Vision:** supported (in-checkpoint tower).
- **Sampling:** SGLang has NO server-wide `--temperature`/`--top-p`/
`--top-k` — sampling rides on the request; the harness sends the Qwen
recipe (temp 0.7, top_p 0.8) via models.yml `compat.extraBody`.
- **Reasoning effort:** the full OpenAI set is accepted (the chat template
has no effort gate).
## Measured
- Boot ~60 s with warm JIT caches (several minutes cold — the first boot
compiles the sgl-kernel + flashinfer kernels; the CUDA 13 toolchain
above is what makes that compile succeed).
- Decode **83.9 tok/s** (no spec decode — nothing to accept).
- Prefill ~2,050–3,370 t/s eager (4-slot shape: a 170,019-token prompt in
50.5 s, ~3,370 t/s; 2-slot shape: a 200,000-token prompt in 97.3 s wall
incl. 8 decode tokens, ~2,056 t/s).
- 4-slot shape: 4 concurrent 54,027-token requests all answered in ~17 s
(216k tokens of live KV at once). 2-slot shape: two concurrent
50,000-token requests both answered (28.8 s wall).
- **Compaction fit (the reslice's purpose):** a 200,000-token prompt —
well past frank-archon's 183,501-token compaction trigger (70% of its
262,144 window) — prefills and completes in ONE request.
- Vision: a 64x64 red square answers "Red" with `image_tokens: 64`.
## Rollback
[[frank-zealot-llama-mimo-q5km]] was the live shape two steps forward; the
practical rollback at the time was [[frank-zealot-llama-mimo-q6k]] (payload at
`/home/dank/models/MiMo-Q6K/`, still on disk 2026-10-18).
This era's own rollback, the llama.cpp Gemma4-12B QAT payload, was staged to
`/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/` on
2026-10-02; its unit/env/serve trio remains in `old/`.
# Citations
- Model: https://huggingface.co/letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound
- SGLang: https://github.com/sgl-project/sglang
- Serve script: `scripts/serve-frank-zealot.sh`; tunables: `scripts/frank-zealot.env`
serverAppConfigs-docs/LLM/frank-zealot-vllm-gemma4.md
-226
View File
@@ -1,226 +0,0 @@
---
type: reference
title: frank-zealot on Dankclaw — vLLM, Gemma4-12B QAT w4a16 (compressed-tensors)
description: "The RETIRED frank-zealot runbook (2026-09-19, same-day revert; rollback target) — vLLM on Dankclaw's RTX 3090 serving google/gemma-4-12B-it-qat-w4a16-ct (Marlin int4 weights, bf16 KV, 262k window) with Google's bf16 MTP assistant, the audio-capable derived image, the no-nvidia-container-toolkit device mounts, measured throughput, and the drafter-Marlin bug that killed the w4a16 assistant. Replaced by [[frank-zealot-gemma4-12b]] on the same day."
tags:
- llm
- frank-zealot
- dankclaw
- vllm
- rtx3090
- gemma
- mtp
- vision
- audio
timestamp: 2026-09-19
---
# frank-zealot — vLLM on the RTX 3090 (Gemma4-12B QAT w4a16)
> **RETIRED 2026-09-19 (same-day revert) — rollback target only in git.**
> This vLLM deployment was cut
> back to [[frank-zealot-gemma4-12b]] (llama.cpp — itself retired
> 2026-10-01; :7771 today runs [[frank-zealot-llama-prism-bonsai2-pq20]])
> the same day it landed. Same port
> (7771), same alias (`frank-small` — the name `frank-zealot` dates from
> 2026-10-06/10-09), same client contract surface — the
> runtime, the model payload, the drafter and the service shape (root
> systemd unit → rootless Quadlet) all changed.
## Identity
| what | value |
|---|---|
| runtime | vLLM `vllm/vllm-openai` dev build `gemma-x86_64-cu129` (base `sha256:e24d3d8e…001e`), audio extra pip-installed into `localhost/vllm/vllm-openai:frank-small` (`8dad46d79c46` — still in rootless storage on the 2026-10-18 `podman images` census; the tag carries the era name) |
| unit | **rootless Quadlet** `dankclaw/frank-small/quadlet/frank-small.container` (era path; deleted by the revert commit `374041c` — root systemd unit retired 2026-09-19) |
| model | `google/gemma-4-12B-it-qat-w4a16-ct` — compressed-tensors w4a16 (int4 Marlin weights, 10,264,229,896 B) |
| model dir | `/home/dank/models/gemma4-12b-qat-w4a16-ct/` (absent from the 2026-10-18 census of `/home/dank/models/` — re-fetch to use this rollback) |
| drafter | `google/gemma-4-12B-it-assistant` **bf16** (845,719,296 B) at `/home/dank/models/gemma4-12b-assistant-bf16/` (absent from the same census) |
| port | 7771 → 8000 (alias `frank-small` in this era; `frank-dragoon` keeps 7770) |
| window | 262,144; KV pool **289,647 tokens** bf16; **2 concurrent full-context sessions** |
| multimodal | native: text + image + **audio** (derived image carries `av`/`scipy`/`soundfile`/`mistral_common[audio]`) |
## Why vLLM
- The w4a16-ct checkpoint is a compressed-tensors format; vLLM's Marlin
kernel is its native int4 GEMM path. llama.cpp could not load it at all.
- Native audio encoder in the checkpoint — the fleet's only audio-capable
endpoint moves here (llama.cpp's path needed the mmproj audio tower).
- CUDA graphs (PIECEWISE 21/21 + FULL 11/11) — the llama.cpp CUDA build on
this host had no graph support.
## The drafter-Marlin bug (why the assistant is bf16)
The w4a16-ct assistant (`kunhunjon/gemma4-12b-qat-assistant-w4a16-ct`)
**cannot draft in vLLM**: the drafter's linear layers are loaded through a
path that does not apply the Marlin kernel, so the packed int4 weights are
misread as bf16 and every draft is garbage — measured acceptance **1.00**
(pure pass-through). There is no "Using MarlinLinearKernel" log line after
the drafter loads. Google's **bf16** assistant (`google/gemma-4-12B-it-assistant`)
has no quantization to miss, and measures **acceptance 2.80–3.56**
(per-position 0.77 / 0.57 / 0.52 / 0.42, ~57 %). The 211 MB delta
(634 → 845 MB) is absorbed by pool rounding: both assistants yield the
identical 289,647-token KV pool.
## KV arithmetic — why two full-context sessions
Gemma4 hybrid: 42 SWA layers (window 1024) + 6 full-attention layers, MHA
16 heads × 512 dim. The SWA layers cache only the 1024-token window, so
per-token KV is dominated by the full-attention layers:
6 × 16 × 512 × 2 (K,V) × 2 B = **196.6 KiB/token** (bf16).
289,647 tokens × 196.6 KiB ≈ 55 GiB of addressable pool… which does not
sound like it fits a 24 GiB card — it fits because vLLM's SWA pool
shares the 1024-window ring across all concurrent requests, so the
marginal cost of a *second* 213k session is ~6 full-attention GiB, not
a second copy of everything.
Measured (salted, 2026-09-19):
| load | KV usage | result |
|---|---|---|
| 1 × 213k | ~35 % | runs |
| 2 × 213k | **69.3 %** | both run in parallel (17.5 % slowdown each) |
| 4 × 213k | — | `Running: 2, Waiting: 2` — the 3rd queues |
**Ceiling: 2 concurrent full-context sessions.** (The llama.cpp
deployment fit 4 slots of 262k with q8 KV; the bf16 KV here is 2× the
bytes, and vLLM's pool arithmetic is coarser. This is the accepted
trade for the native w4a16 format + audio + CUDA graphs.)
## Measured throughput (salted, 2026-09-19)
| probe | tokens | wall | rate |
|---|---|---|---|
| prefill @ 12.7k | 12,706 | 27.2 s | 466 t/s (degraded — concurrent traffic) |
| prefill @ 67k | 67,252 | 52.0 s | **1,293 t/s** |
| prefill @ 213k | 213,044 | 350.3 s | **608 t/s** |
| 2 × 213k prefill | 213,044 each | 374 s batch | 608 → 570 t/s per stream |
| decode @ 213k depth | 300 out | 372 s total | **~13.6 t/s** (bandwidth-bound) |
| decode @ 85k depth | 300 out | 82.3 s total | **~17.6 t/s** |
- **Prefill degrades with depth** (1,293 → 608 t/s): the 6 full-attention
layers re-read the growing KV, and TRITON_ATTN is not as depth-efficient
as llama.cpp's flash-attn path here.
- **Decode at 213k is memory-bandwidth-bound**: ~8.4 GB of KV read per
token against the 3090's ~936 GB/s gives a ~10–14 t/s ceiling; 13.6 is
right at it. At normal depths decode is far faster (the 67k-depth probe
is in the bench log). MTP acceptance ~3.2–3.6 is folded into these
numbers.
- **Bench gotcha (cost an hour)**: a repeated-paragraph filler does NOT
scale linearly in tokens — BPE compresses the repetition harder as the
span grows (1.4M chars ≈ 262k tokens, not the ~175k a linear fit
predicts). Size fillers with the model's own tokenizer
(`AutoTokenizer.encode` on the exact prefix), not char counts.
- `POST /flush_cache` is **404** in this build — salt every probe
(uuid prefix) instead.
## The image
`dankclaw/frank-small/Dockerfile` (era path, deleted by the revert commit
`374041c`; recover with `git show 374041c^:dankclaw/frank-small/Dockerfile`) — 16 comment lines + `FROM
localhost/vllm/vllm-openai:gemma4-unified` + `RUN pip install
--no-cache-dir av scipy soundfile "mistral_common[audio]"`. **No COPY
lines** — the earlier `triton-sm89-sitecustomize.py` hack is gone:
`sm_86` (Ampere GA102) is the *correct* compute capability, confirmed
after the driver swap, so no arch override is needed.
## No nvidia-container-toolkit — manual CUDA mounts
Dankclaw has no nvidia-container-toolkit, so the Quadlet unit wires CUDA
by hand:
- `Device=` for `/dev/nvidia0`, `/dev/nvidiactl`, `/dev/nvidia-uvm`,
`/dev/nvidia-uvm-tools`
- host `libcuda.so.1`, `libcuda.so`, `libnvidia-ml.so.1` mounted read-only
into **both** `/usr/lib/x86_64-linux-gnu/` and `/usr/local/nvidia/lib64/`
- `NVIDIA_VISIBLE_DEVICES=all`, `--ulimit memlock=-1:-1`
If the driver is upgraded on the host, these mount sources move with it
(they are versionless symlinks) — no unit change needed.
## Host driver note
Dankclaw is **Ubuntu 26.04 LTS** (not CachyOS — that is the dev box).
Driver `nvidia-driver-610` **610.57.04** (installed 2026-09-19, replaces
595.91.07). `nvidia-smi` reports compute capability **8.6** — correct for
the GA102/Ampere 3090; the earlier sm_89 confusion was a driver artifact,
not a hardware mystery.
## Deployed config
Repo truth **for this era** (the files carry llama.cpp knobs today): `scripts/frank-zealot.env` + `scripts/serve-frank-zealot.sh`
(mounted into the container at `/scripts/`, which is the entrypoint) +
`dankclaw/frank-small/quadlet/frank-small.container` (era path, deleted at `374041c`). Key values:
`KV_CACHE_DTYPE=auto` (resolves to bf16 — fp8_e5m2 is refused by
compressed-tensors checkpoints and fp8 is hardware-dead on sm_86),
`ENFORCE_EAGER=0` (CUDA graphs on), `max_model_len=262144`, MTP via
`--speculative-config` pointing at the bf16 assistant. `TRITON_OVERRIDE_ARCH`
is **removed**. The serve script refuses to start if the weights, the
assistant, or the vllm binary are missing.
## Client contract
- **Reasoning lives in `message.reasoning`** (vLLM's gemma4 reasoning
parser), not `reasoning_content` — the harness maps it via
`compat.reasoningContentField: "reasoning"` in `~/.omp/agent/models.yml`.
The field is empty when the model emits no thinking tag (behavior, not
a parser bug).
- `preserve_thinking: true`, `temperature`, `top_p` in `extraBody` are
accepted (200, not 400) — vLLM ignores the unknown key.
- Every `reasoning_effort` level accepted (all seven probed 2026-09-17).
- Tools → `tool_calls` (verified `get_weather`).
- Image → 200, described correctly (native vision tower).
- Audio → 200, a 440 Hz WAV tone described correctly (native audio
encoder; verified in the derived image).
- Harness: `contextWindow: 262144`, `maxTokens: 32768`,
`input: [text, image, audio]` — era values; the endpoint reads 200,704 /
16,384 / `[text, image]` today (`~/.omp/agent/models.yml:206,212,214`), and
audio left the fleet with this payload on 2026-10-01.
## Boot ledger
t=135 s to ready (weights are 10.3 GB int4 + 0.85 GB assistant from a
bind mount; Marlin active; CUDA graphs PIECEWISE 21/21 + FULL 11/11;
KV pool 289,647 tokens). Health is 503 until ready — do not kill the
container while it loads.
## Troubleshooting
- **Acceptance 1.00** — the drafter is the w4a16-ct assistant again
(drafter-Marlin bug). Confirm the assistant mount points at
`gemma4-12b-assistant-bf16`.
- **`sm_89` in an error** — stale arch override; `TRITON_OVERRIDE_ARCH`
must be absent. 8.6 is correct.
- **Audio 400/500** — the container is running the *base* image, not the
derived `:frank-small` (missing `av`). In this era, check the deployed image
id is `8dad46d7…` (`podman images`, 2026-10-18: it still exists, unused —
nothing on :7771 runs in podman today).
- **CUDA not visible in container** — a `Device=` line or a lib mount
dropped out of the unit after an edit; the unit is the source of truth,
the host-side generated unit is not.
- Logs: `journalctl --user -u frank-zealot` (rootless Quadlet in this era;
`podman logs` covers only the current `--replace --rm` instance). The unit is
a ROOT unit today: `journalctl -u frank-zealot`.
## Rollback
[[frank-zealot-gemma4-12b]] — the llama.cpp deployment. Its weights,
mmproj and MTP sidecar are **no longer on the host** under
`/home/dank/models/Gemma4-12B-QAT/` (2026-10-18 census: the directory is
gone — it was staged to `.AGENTTRASH/20261002-retired-llm-payloads/` and that
stage dir is gone too), so this rollback re-fetches
`HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` and restores the era
trio from git (`git show 374041c:scripts/frank-small.service`,
`…:scripts/frank-small.env`, `…:scripts/serve-frank-small.sh`); the era quadlet
lived at `dankclaw/frank-small/quadlet/frank-small.container`. One owner per
port: the llama.cpp unit is root, so stop the container unit before starting it.
## References
- `scripts/serve-frank-zealot.sh` — the actual exec line (today: llama.cpp; the
vLLM exec line is in `git show 374041c^:scripts/serve-frank-small.sh`)
- `dankclaw/frank-small/Dockerfile` — the audio-derived image (era path, deleted at `374041c`)
- `dankclaw/frank-small/quadlet/frank-small.container` — the unit (era path, deleted at `374041c`)
- [[frank-zealot-gemma4-12b]] — RETIRED predecessor (rollback)
- [[halogen-flash-server]] — the sibling endpoint on the iGPU
- [[VLLM Overview]] / [[VLLM Index]] — runtime background
serverAppConfigs-docs/LLM/frank-zealot.md
+191
View File
@@ -0,0 +1,191 @@
---
type: reference
title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10777), Ternary-Bonsai-2-9B PQ2_0-MTP, 2 × 262144, exact f16 KV"
description: "The one frank-zealot reference: the live shape on Dankclaw's RTX 3090 (:7771) exactly as scripts/frank-zealot.env ships it, every era with its rollback path, the deduplicated laws of this card, the wire contract, and the owning files."
tags:
- llm
- frank-zealot
- dankclaw
- llama-cpp
- gguf
- rtx3090
timestamp: 2026-10-18
---
# frank-zealot — llama.cpp (PrismML) Bonsai-2-9B PQ2_0-MTP, 2 × 262,144, exact f16 KV
Dankclaw's RTX 3090 endpoint — `http://192.168.0.143:7771`, wire id `frank-zealot`, the fleet's **compaction hub**
(widest window it has). One page per endpoint: eras are rows below, not pages. Siblings: [[frank-archon]],
[[frank-dragoon]]. Deploy with `./scripts/deploy.sh dankclaw --service frank-zealot`; swap mechanics in
[[Model Swap and Rollback]], sizing math in [[Llama Cpp Serving Sizing]]. `scripts/frank-zealot.env` and
`scripts/serve-frank-zealot.sh` are the authority for the shape; where any prose disagrees with them, the prose
is the bug.
## Current shape
| Item | Value |
|---|---|
| Checkpoint | `/home/dank/models/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf` — 2,675,410,720 B, LFS sha256 `c79fcdfaf6b7bf47…` |
| Projector | `…/Bonsai2-9B-MTP/Ternary-Bonsai-2-9B-mmproj-Q8_0.gguf` — 624,229,664 B, sha256 `e0cb6f6988cea323…` — **Q8_0 is mandatory**, not a preference (see Contract) |
| Fetch | `scripts/download-frank-zealot-bonsai9.sh [DEST]` (`--verify-only`; `-f`, byte count, LFS sha256 per file) |
| Runtime | `/home/dank/GitPackages/llama-prism-b10777/llama-server` — PrismML **fork**, `0.2.0-dev (build 10777, commit e3de71522)`; the only loader for this payload |
| Trio | `scripts/frank-zealot.service` (root, relinked, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`, `TimeoutStopSec=180`) · `scripts/frank-zealot.env` · `scripts/serve-frank-zealot.sh` |
| Address / alias | `BIND_ADDR=0.0.0.0`, `LISTEN_PORT=7771`, `ALIAS=frank-zealot` (a flag, not a fact) |
| Flags | `--ctx-size 524288 --parallel 2` ⇒ 262,144 per slot · `--cache-type-k f16 --cache-type-v f16` (exact, **16,384 MiB**) · `--flash-attn on` · `--batch-size 4096 --ubatch-size 2048` · `--n-gpu-layers 99` · `--jinja --reasoning auto --reasoning-format deepseek` · `--metrics` · `--mmproj` · `--log-verbosity 4` · `--presence-penalty 1.0` · **`SPEC_TYPE=none`** |
**Why this shape.** `general.architecture qwen35`: 33 blocks (32 trunk + `blk.32` = the MTP block),
`full_attention_interval 4` ⇒ 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, `nextn_predict_layers 1`
(the head **is in the file**), trained window 262,144, `general.sampling.*` 1.0/0.95/20. Only the 8 full layers bill
per token (32,768 B at exact f16 ⇒ 16,384 MiB over the pool) and the DeltaNet layers add ~50 MiB of recurrent state
**per slot**: exact KV costs the third conversation and nothing else.
**MTP is OFF by measurement, not by absence of a head.** The drafter is *good* — 2026-10-18 measured
acceptance 0.798 / 0.617 per position, mean accepted length 2.41, and the publisher's card says depth 2 works
best — and it still loses here, because the draft context is built against the *target's* pool: `blk.32` bills
4,096 B/position at its default f16 ⇒ 2,048 MiB at 524,288 positions on a trunk that projects 21,234 MiB of the
24,123 MiB addressable (that projection predicts 2,889 MiB free; replace it with the measured pair). Exact KV and a
drafter cannot share this card. `--spec-draft-type-k`/`--spec-draft-type-v` (default f16, verified in this build's
`--help`) would cut the draft to 1,088 MiB — legal on b10777, **never booted beside exact KV**. There is no
`--spec-draft-model`: the MTP draft rides the main model's weights. The env carries the recipe
(`SPEC_DRAFT_N_MAX=2`, `SPEC_DRAFT_N_MIN=0`, `SPEC_DRAFT_P_MIN=0`); `SPEC_TYPE=none` never sends it.
**Harness contract** (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw-3090)`, id `frank-zealot`):
`contextWindow: 262144` — **the per-slot window**, because llama.cpp divides `--ctx-size` across slots and answers a
longer prompt with a 400, never a truncation — plus `maxTokens: 16384`, `input: [text, image]`,
`compat.reasoningContentField: reasoning_content`, `compat.reasoningEffortMap` (**load-bearing**: the template takes
only `xhigh|medium|low` and raises on anything else), `extraBody` 1.0 / 0.95 / 20 / `min_p 0`, and the
`compactionModel` of [[frank-archon]], [[frank-dragoon]] **and** frank-zealot all pointing here.
**The exact-KV boot, measured** (2026-10-18 evening, under the unit, verbosity 4). KV **16,384.00 MiB** =
32,768 B/token × 524,288 positions, to the MiB — rate derivation and ledger agree with no slop, which is what
buries the doc's old "f16 needs 26,112 MiB and cannot boot" figure for good: that was the 27B's 34,816 B/token
applied to the 9B. RS **100.50 MiB** (2 cells, one 50.25 MiB cell lighter than the q8_0 era), compute
1,408.06 + 1,088.07 CUDA_Host, mmproj worst case **868.32 MiB** charged *after* the engine's own fit check,
settled `nvidia-smi` **21,176 used / 2,949 free** of 24,123, `n_slots = 2`, `n_ctx_slot = 262144`,
`DropInPaths` empty, no clamp line, no `cudaMalloc` error — within 24 MiB of the pre-flight projection. That is
a desktop margin: the ~2 GiB of slack is what an image storm has to live inside.
**Rates on this shape are FLOORS, not the shape's rates** — measured the same evening greedy with `ignore_eos`
at 512 tokens, first request discarded, while another consumer drove the other slot (56 new prompts in three
minutes, 96 % GPU-util): the exact condition the bench window exists to exclude, and one contended deep arm
even came back truncated (a 512 KB body accepted as `prompt_n = 4`).
| Arm | q8_0 3-slot record | f16 2-slot, contended |
|---|---|---|
| single-stream decode | 168.7 | **134.1 – 141.5** (145.8 and 163.2 in a lighter minute) |
| concurrent aggregate | ≈300 across 3 (102.8 each) | **256.6 / 262.4 / 265.1** across 2 (≈128–133 each) |
| prefill at ~106k | 2,877 at 106,418 | **3,159.5 at 106,282** |
| decode at 106k depth | never measured | **71.5** |
What the floors already support: per-conversation concurrency **improved** (two streams hold ~130 t/s each where
three held 102.8), endpoint aggregate sits ~12 % under the 3-slot record, prefill beat the record while
contended, and single-stream decode is between parity and −20 %. The 1.882× KV byte bill is real; its true size
on this card is still unmeasured, so quoting 134 or 163 t/s as "the" number is a guess in either direction —
re-run the same arms with the box quiet before a figure is carried anywhere else.
**Fallback rungs**, cheapest first — one line of `frank-zealot.env` each, and each must keep the seam's fit gate green:
| Knob | Effect |
|---|---|
| `UBATCH_SIZE=1024` | hands back 1,306 MiB of compute buffers; prefill −17 %; it is also the arm that ran the drafter at three windows (213.5 t/s) |
| `CACHE_TYPE_V=q8_0` | exact K + quantized V: KV 16,384 → **12,544 MiB**, freeing 3,840 (llama.cpp has no true fp8 KV dtype — q8_0 is the 8-bit class) |
| `CTX_SIZE=401408` | two 200,704 conversations, exact KV at 12,544 MiB (same 3,840 freed) — `contextWindow` MUST follow the per-slot number |
| `CTX_SIZE=786432 PARALLEL=3 CACHE_TYPE_K/V=q8_0` | the era below this one: 13,056 MiB of cache, the third conversation back |
| `CACHE_TYPE_K/V=q4_0` | 10,240 B/token ⇒ 5,120 MiB (the 27B era measured q4_0 at 16,392 used / 7,733 free at 2 × 200,704) |
## Eras
| Window | Payload | Runtime | Shape | Replaced because | Rollback |
|---|---|---|---|---|---|
| 2026-10-18 13:19 → same evening (the boot `68afc6d` replaced) | the same 9B MTP file and projector | PrismML fork 10777 | 3 × 262,144 (pool 786,432), q8_0 K+V = 13,056 MiB, settled 17,956 / 6,169 MiB; 168.7 t/s decode (three runs 168.4–168.7), ~300 t/s over three streams (102.8 each), 2,877 t/s prefill at 106,418 | Frank ordered unquantized KV, and that pool cannot survive it: 24,576 MiB of f16 cache > 24,123 addressable | one env line — `CTX_SIZE=786432 PARALLEL=3 CACHE_TYPE_K/V=q8_0` — and redeploy; no payload move |
| 2026-10-18, 24 min (arm under the unit) | `Bonsai2-PTQ1_0/Ternary-Bonsai-2-27B-PTQ1_0.gguf` 5,946,648,928 B + hardlinked Q8 mmproj | PrismML fork 10777 | 2 × 200,704, batch/ubatch 3072/1536 | Frank scratched the 27B plan — ~48 t/s decode vs 76.6 for PQ2_0 on the same weights | `/home/dank/models/Bonsai2-PTQ1_0/` + `scripts/download-frank-zealot-bonsai2-ptq1.sh` |
| 2026-10-17 → 10-18 | Ternary-Bonsai-2-**27B** PQ2_0 7,206,168,928 B `3907dc16…` + Q8_0 mmproj 629,246,976 B | PrismML fork **10754** (`~/GitPackages/llama-prism/`) | 2 × 200,704 (pool 401,408), q8_0 K+V = 13,328 MiB, settled 22,664 / 1,461; 76.6 t/s decode, 109.4 aggregate over two streams, 691.7 t/s prefill at 166,027; MTP impossible — no head in the file | the 9B bills half the KV per token (8 full layers, not 16), which funded three slots at the native window and beat it on every rate | `/home/dank/models/Bonsai2-PQ2_0/` + the 10754 fork tree, `scripts/download-frank-zealot-bonsai2.sh`, revert the swap commit |
| 2026-10-07 → 10-17 | Qwen3.5-9B-Uncensored-HauhauCS-Aggressive Q6_K 7,359,259,008 B `c0ba7beb…` + BF16 mmproj 921,704,448 B | llama.cpp master `5fc4f3c` CUDA sm_86 (`~/GitPackages/llama-master/build-cuda/`) | 2 × 200,704, exact f16 K+V, MTP off (no `nextn` tensors) | the ternary 27B bought a larger model class at the same window | `/home/dank/models/Qwen35-9B-HauhauCS-Q6K/` + the master build, `scripts/download-frank-zealot.sh` |
| 2026-10-07 (same day) | Gemma4-26B-A4B-QAT Q4_K_M 16,796,015,520 B + BF16 mmproj 1,194,827,776 B + unused `gemma4-assistant` MTP 251,937,728 B | llama.cpp master CUDA | 1 × 200,704, f16 K + q8_0 V, ubatch 1024, MTP off; 21,868 / 2,257 settled, 139.9 t/s decode (the ~4 B-active MoE is at home here); MTP ON would cost ~1.9 GiB and boot at 319 MiB free | ONE slot serialized compaction; the 9B's lighter weights bought two slots and exact KV | `/home/dank/models/Gemma4-26B-A4B/` (ungated — plain `curl -fL`) |
| 2026-10-03 → 10-05 | MiMo-V2.6-Distill-Qwen-9B Q6_K 7,793,710,624 B `ef96d05a…` + f16 mmproj 918,166,048 B | llama.cpp master CUDA | 2 × 200,704, f16 K+V, batch 4096 / ubatch 2048; 21,846 / 2,279 settled, 94.44 t/s decode | Frank ordered the uncensored 27B with MTP drafted | `/home/dank/models/MiMo-Q6K/` |
| 2026-10-05 (committed `77decb5`→`380aafa`, **never deployed**) | Ornith-1.5-9B Q8_0 | llama.cpp master CUDA | 1 × 262,144, f16 K+V | Swift cut over the same evening while this deploy sat blocked at the whole-tree docs gate | `/home/dank/models/Ornith-Q8/`; revert to `380aafa`'s tree |
| 2026-10-05 → 10-07 | Swift-1.5-27B-Uncensored RCO-IQ3_XXS-mtp 10,442,827,392 B `e13d3e74…` + publisher `mmproj-BF16.gguf` 931,145,888 B | llama.cpp master CUDA | 2 × 114,688, q8_0 K+V, `draft-mtp` depth 3, ubatch 2048; 23,120 / 1,005 settled (489 MiB at the 2 × 131,072 shape — the tightest fit ever here), 63.3 t/s at the recipe temp, 72.9 at temp 0, 48.0 with the drafter off | the 9B Q6_K rung returned the 200k window and exact KV | `/home/dank/models/Swift15-UncIQ3XXS/`; the IQ3_S-mtp rung at `/home/dank/models/Swift15-IQ3S/`; commits `da36fa3` → `f2575a1` → `d8da7a9` |
| 2026-10-02 → 10-03 | MiMo Q5_K_M 6,876,124,704 B `04cf46f2…` + the same f16 mmproj (hardlinked) | llama.cpp master CUDA | 2 × 200,704, f16 K+V, batch 2048 / ubatch 1024; 20,530 used / 4,046 apparent (true `free` 3,571), 104.84 t/s decode | the ~3.5 GiB it left free bought the Q6_K rung | `/home/dank/models/MiMo-Q5KM/` |
| 2026-10-01 → 10-02 | MiMo Q6_K (same file as above) | llama.cpp master CUDA | 2 × 262,144, f16 K + q8_0 V | one night: the Q5 rung bought the 2–3 GiB headroom band back | `/home/dank/models/MiMo-Q6K/` (same dir) |
| 2026-10-01 (hours) | MiMo-V2.6 INT8-W8A16-AutoRound, 8 shards 12,891,718,440 B | SGLang in venv `~/GitPackages/sglang-cu128` (torch cu130 + an in-venv CUDA 13.0 nvcc) | 2 × 262,144 over a pinned 606,208 pool, `fp8_e4m3` KV, NEXTN off; 83.9 t/s decode, ~2,050–3,370 t/s eager prefill, ~60 s boot warm | llama.cpp was the only runtime exposing asymmetric per-tensor KV | payload purged — re-fetch `old/download-frank-zealot-sglang-int8.sh`; the venv is still on disk |
| 2026-09-17 → 10-01 | Gemma4-12B-QAT-Uncensored Q4_K_M 6.87 GiB + BF16 mmproj (it carried an **audio** encoder) + `mtp-gemma-4-12B-it.gguf` sidecar 0.24 GiB | llama.cpp master `37b53fd` CUDA with `-DGGML_CUDA_FA_ALL_QUANTS=ON` (`~/GitPackages/llama.cpp-mainline/build-cuda/`) | 4 × 262,144, q8_0 K+V, `draft-mtp,ngram-mod` depth 3 / poll 50; idle 22,346–23,300 MiB, ~965 t/s prefill and 257–302 t/s decode at 254k depth; each extra full-context slot costs 3,450 MiB so 4 is the max | the MiMo GGUF path (qwen35 hybrid, asymmetric KV) took the card | re-fetch `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced`; era trio in `git show 374041c:scripts/frank-small.service` / `.env` / `serve-frank-small.sh` |
| 2026-09-19 (same-day revert) | `gemma-4-12B-it-qat-w4a16-ct` 10,264,229,896 B + **bf16** assistant 845,719,296 B | vLLM dev image `localhost/vllm/vllm-openai:frank-small` (`8dad46d79c46`, still in rootless storage), rootless Quadlet | 262,144 window, bf16 KV pool 289,647 tokens, 2 concurrent sessions (the 3rd queues), image + audio; 1,293 t/s prefill @67k falling to 608 @213k, ~13.6 t/s decode @213k depth, MTP acceptance 2.80–3.56, boot 135 s | reverted to llama.cpp the same day | both payloads purged; `git show 374041c^:dankclaw/frank-small/Dockerfile` and `…:dankclaw/frank-small/quadlet/frank-small.container` |
| 2026-09-16 → 09-17 | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S + BF16 projector, in-file MTP drafted | mainline llama.cpp | single-slot 3090 shape | superseded by the Gemma4-12B payload | re-download from ISTA-DASLab — no payload and no fetch script survive |
| 2026-09-02 → 09-15 (**Dankserver**, `:7770`, the card's previous host) | `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` ~35 GB (before it official `Qwen3.6-35B-A3B-FP8`) | FreeToken v0.1.2 (`ft`), venv `~/freetoken/.venv` | MoE cache 2,048 slots + KV 453,005 tokens + 24 mamba-state slots; ~30 t/s single-stream decode, ~87 t/s at batch 4, ~1,130 t/s prefill at 4k | the 3090 left Dankserver and `frank-medium-rocmfpx` took the endpoint and port | host units `~/.AGENTTRASH/20260915-frank-zealot-retire/host-units/`, repo trio in `scripts/.AGENTTRASH/` |
| ~2026-08 (Dankserver 3090, pre-FreeToken) | Qwen3.6-35B-A3B Q6_K_P, then Tiel-Coder-35B-A3B UD-Q6_K_XL, with an Ornith Q6_K rung | llama.cpp `555881ebc` (b10121) CUDA | `--n-cpu-moe 38` deployed (36 recommended), batch 4096 / ubatch 2048, 700,416-token f16 K+V pool (800,768 cells did not fit); ~45.5 t/s decode and 1,188 t/s prefill at `ncmoe 35` | FreeToken kept experts in host RAM instead of CPU-offloading layers | re-download; nothing from this rung survives on disk |
## Laws learned here
- **Addressable ≠ advertised.** The card is 24,576 MiB but leaves **24,123 MiB**: quote `nvidia-smi`'s `free` field, never `total − used` — the driver's reservation is the difference.
- **Exact KV and a 786,432 pool cannot coexist here**: 32,768 B/token ⇒ 24,576 MiB before a single weight loads (the retired "26,112 MiB" note doubled q8_0's scale overhead); the 27B analogue died at **25,088 MiB** with `cudaMalloc failed: out of memory`. Halving the pool funds exact KV; the per-slot window stops at the trained 262,144 (no YaRN here), so a third conversation is a KV-dtype decision, never more context.
- **Exact KV reads 1.882× the bytes per generated token, not 2×** (32,768 vs 17,408 B/token, 2026-10-18): a decode drop of that order is EXPECTED and must be measured, while prefill should hold — and this service can afford the dtype precisely because its load is the compaction hub's ~170k-token prefill, which is weight-bandwidth bound and never re-reads the cache per generated token. The dtype is paid in decode and concurrency.
- Never `--fit` with `parallel > 1` — it maximizes the *total* pool and shrinks each slot: set `CTX_SIZE` explicitly and keep it divisible by `PARALLEL` (the seam enforces it).
- **KV rate law.** `full_layers × 4 KV heads × 256 dim` = **8,192 elements per token per cache** on this arch (the engine prints `n_embd_k_gqa = n_embd_v_gqa = 1024`), ×2 B (f16), ×1.0625 (q8_0 — 32 codes plus one 2-byte scale
per 32-block), ×0.625 (q4_0): **32,768 / 17,408 / 10,240 B/token** on this 8-full-layer 9B, and 34,816 for a
16-full-layer 27B at q8_0. Everything else bills separately: DeltaNet recurrent state per **slot** (150.75 MiB for
3 cells, and an armed depth-2 drafter triples it to 452.25), compute buffers at graph capture, the mmproj after the
fit. On gemma4-class hybrids `attention_k_eq_v` does **not** shrink the allocation — the cache reserves the K/V pair
twice, 6.125 B/elem at f16 K + q8_0 V.
- **`--flash-attn on` is pinned, not decorative**: a quantized V exists only on the FA path and the engine falls back silently without it. llama.cpp has **no true fp8 KV dtype** — q8_0 is the 8-bit class (1.0625 B/elem, ~8.5 effective bits). `GGML_CUDA_FA_ALL_QUANTS` is vestigial and build-specific: the PrismML fork carries it ON (since 2026-09-17) while the master build had it OFF and still dispatched Q4/Q5 KV unconditionally (`ggml-cuda/fattn.cu` has no guard), so quantized KV never required it.
- **Prices come from boots, not formulas, and an arm belongs to the payload it was measured on**: 3 × 180,224 +
q8_0 V fits with Q5_K_M (21,684 used / 2,441 free) and does **not** fit with Q6_K inside the 2–3 GiB band. The
mmproj worst case (~870 MiB) is printed before fitting but never billed by the engine's own fit check. Only
`tests/frank_zealot_unit.bats`'s fit gate can see a boot that would die at `cudaMalloc`, so every retune keeps it
green — and the first verbosity-4 ledger plus settled `nvidia-smi` of a new dtype belongs on this page, because
projections drift.
- **Ubatch is a prefill knob.** Decode never moves with it and the logical `--batch-size` is neutral; prefill climbs monotonically to **2048** on this CUDA card (+4.6 % at 8.9k depth; 710 → 1,188 t/s), a prompt shorter than the ubatch cannot see the knob at all (the control row), and a verdict never crosses backends — the 2× R9700 split peaks at 512. Each rung costs ~0.6–0.75 GiB of compute buffers; 2048 is the ceiling for a 27B-on-3090 shape.
- **A quant rung is paid in decode**: Q5_K_M → Q6_K is +875 MiB of weights and **−10 %** rate on this bandwidth-bound card, and the byte-bandwidth law predicts a rung's rate within 1 %.
- **Speculation pays only with room on the card.** The 2026-10-18 arm census (all q8_0 K+V at the 786,432 pool): 3 × 262,144 spec-OFF ubatch 2048 → 17,876 / 6,249 free, 167.2 t/s; spec-ON ubatch 2048 → 22,706 / 1,419, 109.2 t/s; spec-ON ubatch 1024 → 21,400 / 2,725, 213.5 t/s; 2 × 262,144 spec-ON → 213.2; 1 × 262,144 spec-ON → 214.2. That is **+28 % decode whenever the card has room**, and the doors back are one window per conversation, or q8_0 KV with ubatch 1024. The draft's own bill at that shape was 3,072.00 MiB of KV plus ~1,758 MiB of buffers.
- Draft KV is built against the target's **whole** pool (~2.4 GiB measured on the 27B Swift boot, not the ~0.4 GiB a proposal-depth estimate suggests). Spec flags move as a SET — `SPEC_TYPE=none` strips all of them and half a set has segfaulted at load. Two context-backed drafters cannot share a process, so `draft-mtp,ngram-mod` is the only legal pair, and on a qwen35 hybrid ngram-mod disables recurrent-state rollback so rejected drafts degrade silently.
- Acceptance alone never justifies speculation: 2026-08-25 measured 26.8–35.2 t/s drafting against 53.7 no-spec.
- A checkpoint declaring `*_num_hidden_layers` MTP while carrying zero `mtp.*`/`nextn.*` tensors boots a 4.24 GiB ghost drafter at `accept rate: 0.00` — read the safetensors index, not the config.
- MTP is a property of the **file**: the plain PQ2_0 sibling could not draft at any flag value; only the `-MTP` export carries `blk.N.nextn.*`.
- **The format is the runtime lock.** PQ2_0 is `general.file_type 141` over 202 tensors of custom GGML type **142** plus `prism.hadamard.*` metadata: stock llama.cpp refuses the type, and a loader ignoring the Hadamard transform would dequant **wrong, silently** (same lock class as ROCmFPX). Build ≥ b10776 is additionally the floor that pairs MTP with a quantized V cache.
- Check arch support in the source tree: `grep -c` on the packed binary returns 0 for every arch, including the one it serves.
- **Base alignment.** Bonsai-2 is the censored base lineage. `/home/dank/models/Bonsai2-CRACK/` matches the **27B** geometry (a one-line `MODEL_FILE` swap there, identical tensor census); no dealigned 9B exists, so for this payload the refusal escape hatch is a fetch, not a revert.
- **A stale bench drop-in hijacks the production boot and no health check sees it.** An `Environment=FRANK_ZEALOT_ENV_FILE=/tmp/<arm>.env` in `frank-zealot.service.d/` booted a 27B PTQ1_0 payload while `/health` answered 200, `/v1/models` echoed the alias flag and the e2e gate passed 33/33 — none of them read the payload.
- After any bench window: `systemctl show -p DropInPaths` → `tr '\0' ' ' </proc/$pid/cmdline` → ledger, in that order; drop-ins retire to `~/.AGENTTRASH/<date>-<topic>/`, and payload decisions land on this page the moment they are made, or an approved arm becomes indistinguishable from a rogue one a day later.
- **An arm env must use the serve script's own variable names** (`CACHE_TYPE_K`/`CACHE_TYPE_V`, `BIND_ADDR`, `NGL`): an invented `KV_CACHE_TYPE` is swallowed by `: "${CACHE_TYPE_K:=f16}"` and the arm boots f16 — an OOM that reads like a model fault.
- **Stop, shebang, GPU node.** llama.cpp rides out the default stop timeout (`TimeoutStopSec=180` is on the unit for that reason) and a hand-booted arm keeps answering 7771 behind a `failed` unit — kill it **by PID**, never `pkill -f llama-server`.
- `serve-frank-zealot.sh` is a direct `ExecStart`, so a missing shebang is a `203/EXEC` boot loop at `Mem peak: 1.9M` with the GPU untouched: `head -1` is a deploy check.
- `No CUDA GPUs are available` ⇒ `nvidia-modprobe -u -c=0`; KV spilled into unified memory halves decode silently, so re-read `nvidia-smi` after any shape change.
## Contract
- `/health` → `{"status":"ok"}`; `/props` reports `n_ctx` and `presence_penalty`; `/slots` shows occupancy; a third
concurrent conversation **waits** for a slot — the slot count is the concurrency contract.
- `/metrics` answers 200 with 15 `llamacpp:*` families **only with `--metrics`**; without it llama.cpp answers **501**, which Prometheus reads as DOWN while `/health` stays 200. The scrape target is `host.containers.internal:7771/metrics` and the oldest retained sample of `up{job="frank-zealot"}` (2026-10-11) is already 0 — this target was never successfully scraped, across every payload era. `METRICS=0` is the documented opt-out and re-breaks it; the counters cost nothing on the request path.
- Reasoning arrives in **`reasoning_content`** (`--reasoning-format deepseek`) and **counts against `max_tokens`**: a short budget yields empty `content` with `finish_reason: "length"`, which reads as a broken model, not a mid-thought one. The slot prefix cache is live (`cached_tokens` in `usage`).
- A client that sends no sampler is **not** greedy — the engine applies the payload's own `general.sampling.*`
(1.0 / 0.95 / 20). `presence_penalty` is the one recipe value with no GGUF metadata key, so it must reach argv;
the serve script uses `${PRESENCE_PENALTY=1.0}`, deliberately **not** `:=`, because `""` is the documented opt-out.
- Vision: only the OpenAI **`image_url`** content part reaches the model — the legacy top-level `images` field bumps `usage.prompt_tokens` while the model invents an answer. Probe with a **≥ 256 px** solid color (a postage stamp answers "video") and budget **≥ 512** tokens; a 512 px image bills ~280 prompt tokens.
- **Q8_0 mmproj, not BF16**: the publisher warns the BF16 tower "can crash llama-server when several image requests run at once", which is exactly the `--parallel 2` shape. `MMPROJ_FILE=` (explicitly empty) is the text-only opt-out and returns ~1.4 GiB; a path at a missing file is a boot **refusal**, never silent blindness.
- Tools: native via `--jinja`; `finish_reason: tool_calls` with valid JSON arguments.
- Two non-faults: a `400` naming 262,144 is one conversation outgrowing its **slot**, not pool exhaustion; and
the raw `/completion` path is a **dead instrument on this fork build** — a greedy `ignore_eos` run answers
**500 "The model produced output that does not match the expected Content-only format"**, because build 10777
guards the raw path the moment reasoning markup leaks (older builds only logged `failed to parse reasoning
content` and answered anyway). The honest per-request instrument here is **`/v1/chat/completions`**, whose
non-stream response carries the same `timings` object (`prompt_n`, `prompt_per_second`, `predicted_n`,
`predicted_per_second`). Keep `VERBOSITY=4` — 3 is the engine default and cannot prove the KV or draft sizes,
while levels above 4 flood the journal.
- llama.cpp does **not** validate the requested model id: it answers and echoes the served `ALIAS`, so a rename breaks id-keyed consumers (the OpenWebUI `model` row, its `access_grant`, `openai.api_configs`) and nothing server-side. Alias history: `frank-small` → `frank-secondary` (2026-10-06) → `frank-zealot` (`7c77326`, 2026-10-09); the service and unit took the name 2026-10-10 (`55f87fc`).
- The first request after a boot is a warmup artifact (37 t/s class) and is never a measurement; never time a wall-clock aggregate on a generation that stops on its own (use `ignore_eos`).
## Files
- `scripts/frank-zealot.env` — **the authority for the values above** and nothing else: as of 2026-10-18 it carries
the knobs plus the one-line reason a reader editing that line cannot recover here. The arm table, the reference
boot ledger and the KV derivation all live on this page now; the header used to hold them, and two of its
rung figures were wrong (`CACHE_TYPE_V=q8_0` and `CTX_SIZE=401408` both free **3,840 MiB** at the 524,288 pool,
not 8,192 and not ~6.7 GiB — the Fallback rungs table above is the corrected statement).
- `scripts/serve-frank-zealot.sh` — the exec line and its rationale comments; honors `FRANK_ZEALOT_ENV_FILE` for arm boots; exits 1 naming a missing binary, GGUF or projector rather than half-loading (a restart policy would turn an unfinished download into a crash loop that reads like a runtime fault).
- `scripts/frank-zealot.service` — root unit, relinked into `/etc/systemd/system/`.
- `scripts/download-frank-zealot-bonsai9.sh` (active) · `…-bonsai2.sh` · `…-bonsai2-ptq1.sh` ·
`scripts/download-frank-zealot.sh` (master-era GGUFs) · `old/download-frank-zealot-sglang-int8.sh` (SGLang).
- `tests/frank_zealot_unit.bats` — stub-argv seam plus the VRAM fit gate.
- `scripts/deploy-remote.sh` — `deploy_llm_zealot` (payload-path gates, MainPID-hardened health); see [[deployment]].
- `shared/prometheus/prometheus.yml`, `shared/prometheus/rules/llm-fleet.yml`,
`shared/grafana/provisioning/dashboards/llm-fleet.json` — the scrape, the alert, the dashboard.
- Live state, not repo files: the OpenWebUI picker row (back up `webui.db` before re-keying) and `~/.omp/agent/models.yml`.
serverAppConfigs-docs/LLM/freetoken-benchmark-sweep.md
+1 -1
View File
@@ -90,5 +90,5 @@ Tiel-Coder finetune.
## See Also
- [[freetoken-frank-small]] — production runbook
- [[frank-zealot]] — production runbook
- [[dealign-ornith-crack]] — the prior llama.cpp deployment (retired)
serverAppConfigs-docs/LLM/freetoken-frank-small.md
-224
View File
@@ -1,224 +0,0 @@
---
type: configuration
title: FreeToken frank-zealot Runbook
description: RETIRED 2026-09-15 - FreeToken v0.1.2 serving batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 as frank-zealot on Dankserver (RTX 3090 era).
tags:
- freetoken
- dankserver
- frank-zealot
- qwen3.6
- moe
- fp8
timestamp: 2026-08-30
---
# FreeToken frank-zealot Runbook
> **RETIRED 2026-09-15.** The RTX 3090 left Dankserver and the successor
> [[frank-medium-rocmfpx]] (charlie12345/ROCmFPX llama.cpp on 2x Radeon AI
> PRO R9700) took the :7770 endpoint and the port. This page is kept as the
> retirement record of the FreeToken/CUDA era; the unit was archived to
> `~/.AGENTTRASH/20260915-frank-zealot-retire/host-units/` and the repo files
> to `scripts/.AGENTTRASH/`.
The `frank-zealot` service on Dankserver is served by
**FreeToken v0.1.2** — an edge-native MoE serving engine that keeps
expert weights in host RAM with an LRU GPU cache. Model history on this
runtime: official `Qwen/Qwen3.6-35B-A3B-FP8` (since 2026-09-02) →
`batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` (2026-08-30,
same format and pool budget; re-landed 2026-09-01 after two
compressed-tensors FP8 candidates failed to load — see Quantization
Compatibility). The engine itself replaced the llama.cpp
Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment.
## Model
| Field | Value |
|-------|-------|
| HuggingFace | `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` |
| Lineage | Qwen/Qwen3.6-35B-A3B → huihui-ai abliterated → this FP8 (quantizer batsclamp) |
| Architecture | `qwen3_5_moe` (Qwen3_5MoeForConditionalGeneration) |
| Quant | native modelopt block-fp8 e4m3, dynamic activations, 128×128 `weight_block_size`, bf16 `weight_scale_inv` — byte-format identical to the official FP8 repo |
| BF16 kept | lm_head, embed_tokens, all norms, router gates, mamba state params, visual encoder, MTP module |
| Experts | un-fused per-expert layout (`mlp.experts.{0..255}.*`) matching official Qwen FP8 — same FreeToken loader path |
| Size on disk | ~35 GB (26 safetensors shards) |
| Path | `/home/dank/models/huihui-qwen36-35b-a3b-abliterated-fp8` |
| Layers | 40 (10 full-attention + 30 GDN/linear-attention) |
| Experts | 256 per layer, 8 used per token |
| KV heads | 2 × 256 dim |
| Context | 262,144 (model max) |
| Chat template | froggeric/Qwen-Fixed-Chat-Templates (fixed Qwen3.6 template, shipped in `tokenizer_config.json`; FreeToken reads it from the checkpoint — no override) |
**Important:** this is the Huihui *abliterated* finetune (refusals
removed) — not the base Qwen weights served before 2026-08-30, and not
the earlier Tiel-Coder finetune. Expect a direct-answers assistant.
## Runtime
| Field | Value |
|-------|-------|
| Engine | FreeToken v0.1.2 (`ft` CLI) |
| Venv | `/home/dank/freetoken/.venv` (uv-managed CPython 3.12) |
| Dependencies | torch 2.11.0+cu130, flashinfer 0.6.17, apache-tvm-ffi |
| CUDA toolchain | pip-installed nvidia-cuda-nvcc/crt/nvvm/runtime 13.2.86 |
| API | OpenAI-compatible (`/v1/chat/completions`, `/v1/models`) |
## Pool Configuration
The GPU budget (~17.6 GiB after weights + fixed caches) is split
MoE-first between expert residency and KV pages:
| Pool | Size | VRAM | Notes |
|------|------|------|-------|
| MoE cache | 2,048 slots (20%) | 6.0 GiB | LRU expert cache, out of 10,240 total |
| KV cache | 453,005 tokens | 8.6 GiB | ~20 KiB/token (10 attn layers only) |
| Mamba state | 24 slots | 1.4 GiB | Per-sequence GDN state (caps concurrency) |
### Live Resize
KV and MoE can be re-partitioned without restart:
```bash
ft ctl --base-url http://127.0.0.1:7770 cache --kv N --moe M --wait 300
```
- **Growing KV** (shrinking MoE): works live.
- **Shrinking KV** (growing MoE): requires restart (old + new pools
cannot coexist in VRAM during the swap → OOM).
- The fit check refuses overcommit: if the plan doesn't fit, the old
cache is retained and serving continues.
### Benchmark Results (2026-09-02, base FP8 checkpoint)
See [[freetoken-benchmark-sweep]] for the full bs1/bs3/bs4/bs6 sweep
across KV pool sizes 8k–700k.
At the production config (20% residency, bs4):
| Metric | Value |
|--------|-------|
| bs1 decode | ~30 tok/s |
| bs4 aggregate | ~87 tok/s |
| bs4 per-request | ~22 tok/s |
| Prefill (4k) | ~1,130 tok/s |
## Quantization Compatibility
FreeToken v0.1.2 supports these safetensor quants for `qwen3_5_moe`:
| Format | Detection | Works |
|--------|-----------|-------|
| BF16 (no quant config) | default | ✅ (experts too large for 60 GB RAM) |
| modelopt block-fp8 (128×128) | `quant_method: "fp8"` + `weight_block_size` | ✅ (current) |
| NVFP4 (modelopt) | `quant_algo: "NVFP4"` | ✅ (best for VRAM) |
| NVFP4 (compressed-tensors) | 4-bit float, group_size=16, strategy=tensor_group | ✅ |
| Per-tensor FP8 (modelopt) | scalar `weight_scale` | ✅ |
| Per-channel FP8 (compressed-tensors) | `weight_scale: [O, 1]` | ❌ loader bug |
| Block-fp8 128×128 (compressed-tensors) | `quant_method: "compressed-tensors"` + `block_structure: [128, 128]` | ❌ dequant shape bug |
| Quark W8A8 INT8 | per-channel scales | ❌ loader bug |
| GGUF | only `gemma4` arch | ❌ for qwen3_5_moe |
**2026-09-01 live verification (Dankserver):** both compressed-tensors
FP8 shapes fail on FreeToken v0.1.2, each a clean exit at load (37 GB
fits RAM — no swap-thrash). `ornith-ai/Ornith-1.5-35B-A3B-FP8`
(per-channel) aborts in expert-source resolution:
`ValueError: Missing MoE expert source layers: {'gate_up': ...}` — its
expert tensors do not follow the `mlp.experts.{N}.gate_proj/up_proj/
down_proj` naming the MoE provider scans for.
`lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8` (block
128×128, standard naming) resolves experts but aborts in dequant:
`RuntimeError: The size of tensor a (2048) must match the size of
tensor b (...)` in `_dequant_fp8_weight`, which multiplies
`weight × weight_scale` directly instead of expanding the block scales.
modelopt block-fp8 remains the only FP8 path that loads.
## Sampling
**frank-zealot samples by default, and our config never said so.**
`scripts/serve-frank-zealot.sh` passes no sampling arguments at all — only
`--model`, `--served-model-name`, `--host`, `--port`, `--moe-cache-size`,
`--num-tokens` and `--max-running-requests`. The behaviour comes from the
runtime's own default:
```python
# freetoken/server/args.py:31
sampling_defaults: str = "model"
```
`--sampling-defaults model` fills `temperature` / `top_k` / `top_p` from the
**checkpoint's** `generation_config.json` for any request that omits them.
This checkpoint ships:
```json
{ "do_sample": true, "temperature": 1.0, "top_k": 20, "top_p": 0.95 }
```
Verified behaviourally 2026-09-12 — three identical requests with **no
sampling parameters sent at all** produced three different reasoning traces
(sha `13ca02…`/487 chars, `d49928…`/540, `7b81b8…`/577). Greedy decoding
would be byte-identical.
### This is the opposite of frank-dragoon
| | frank-zealot (FreeToken) | frank-dragoon (halogen) |
|---|---|---|
| Silent client (no `temperature`) | **samples** at temp 1.0 / top_k 20 / top_p 0.95 | **greedy** — no server-side sampler defaults |
| Deterministic across runs | No | Yes |
| Where the default comes from | checkpoint `generation_config.json` | none; the request decides |
A client that talks to both endpoints and sends no sampler gets
non-deterministic answers from :7770 on Dankserver and reproducible ones from
:7770 on Dankclaw. If a workflow needs determinism from frank-zealot it must
send `temperature: 0` explicitly, or the service must be relaunched with
`--sampling-defaults none` (framework defaults only).
### Related request behaviour
- **Thinking consumes the budget.** At `max_tokens: 200` all three probe runs
returned `finish_reason: "length"` with **empty `content`** and the whole
budget spent in `reasoning_content`. Budget for thinking, or the answer
never appears.
- **The API does not validate unknown fields** — a request carrying
`totally_bogus_field: 123` returns 200, as does
`chat_template_kwargs: {"preserve_thinking": true}`. A 200 is not evidence
that a key was understood.
## Files
| File | Purpose |
|------|---------|
| `scripts/serve-frank-zealot.sh` | Serve script (launches `ft serve`) |
| `scripts/frank-zealot.env` | Tunables (model path, pool sizes) |
| `scripts/frank-zealot.service` | Systemd unit |
## Limitations
- **Model identity**: serves the Huihui abliterated FP8 finetune, not the Tiel-Coder coder finetune from the llama.cpp era.
- **JIT compilation**: first startup compiles flashinfer/tvm-ffi kernels
(~5–15 min cold). Cached afterward in `~/.cache/flashinfer` and
`~/.cache/tvm-ffi`.
- **CUDA toolchain**: driver-only install (no system CUDA toolkit). The
venv provides pip-installed nvcc 13.2.86 for JIT compilation. All
nvidia-cu13 packages must be at the same version.
- **Env var collisions**: never name tunables `TEMP`, `HOST`, or
`GPU_MAX_HEAP_SIZE` (HIP runtime reads these).
## Operational quirks
- **`model` field is required** in `/v1/chat/completions` bodies —
FreeToken rejects requests without it with a 422 (`Field required`),
unlike llama.cpp's server which defaults to the loaded model. Send
`"model": "frank-zealot"`.
- Load on this swap took well under the old llama.cpp numbers:
weights → experts (parallel build) → CUDA graph capture → Uvicorn
in ~2 minutes; the 2026-08-30 swap verified ~40 tok/s decode on a
282-token completion.
## Citations
- FreeToken paper: arXiv 2608.16157
- FreeToken PyPI: `freetoken[accel]` v0.1.2
- HuggingFace model: `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8`
- Upstream weights: `huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated`, `Qwen/Qwen3.6-35B-A3B-FP8`
- Fixed chat template: `froggeric/Qwen-Fixed-Chat-Templates`
serverAppConfigs-docs/LLM/halogen-flash-server.md
-478
View File
@@ -1,478 +0,0 @@
---
type: reference
title: halogen-flash-server on Dankclaw (frank-dragoon)
description: The active frank-dragoon runbook — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing.
tags:
- llm
- halogen
- dankclaw
- frank-dragoon
- strix-halo
- gfx1151
- qwen4exp
- rocm
timestamp: 2026-09-13
---
# halogen-flash-server on Dankclaw (frank-dragoon)
> **ACTIVE again since 2026-09-16, and retuned.** It replaced the llama.cpp /
> kingjones30-ROCmFPX STRIX deployment on 2026-09-11 (see
> [[kingjones-strix-rocmfp4]]), was displaced for two days by ik_llama.cpp (see
> [[frank-dragoon-ik-uncensored]], the current rollback), and came back as the
> shape the fleet asked for: two 200k-token conversations rather than the
> native-context pair. Deployed as a rootless Quadlet container,
> `dankclaw/frank-dragoon/quadlet/frank-dragoon.container`, port 7770. The RTX 3090
> it does not touch now belongs to [[frank-zealot-qwen38-gsq-rco]].
The engine is a from-scratch serving runtime for the Qwen3.8-Flash-Next
architecture on AMD's gfx1151 (Strix Halo) — the same silicon this host runs.
Upstream's own comparison table measures it at **roughly 4x the prefill
throughput** of the llama.cpp-family runtimes it ships against (EngramHalo.cpp,
ROCmFP4, CIRU-IU4) on a 32K prompt, and prefill is most of the wall clock on any
prompt with real context. That, not decode, is the reason for the swap.
## Shape
One container runs both halves of the stack:
| half | port | reachable |
|---|---|---|
| engine | 8730 | container loopback only — **unauthenticated, never publish it** |
| OpenAI front-end | 8731 | published as host **7770** |
The `all` mode is upstream's default and the shape a single-container host
wants: the front-end talks to the engine over loopback inside the container, so
there is one process tree, one log, and one published port.
`HALOGEN_MODEL_ID=frank-dragoon` sets the id reported at `/v1/models`, in
`/health`, and in every response body. Requests are **never rejected for naming
a different model** — the server serves whatever asks. Keeping the fleet alias
is therefore free, and it means Hermes, the harness config, the OpenWebUI
discovery, the `services.json` probe and the docs needed no rename.
## Weights
`peonist-ai/halogen-qwen3.8-flash-next` (HuggingFace, public and ungated):
| file | size | role |
|---|---|---|
| `qwen38-flash-next-w4b.hgn` | 115.55 GiB | the checkpoint |
| `qwen38-flash-next-w4b.overlay.hgn` | 2.40 GiB | **quality sidecar** — picked up automatically; the engine warns at startup if it is missing. **Refreshed 2026-09-16**: the sidecars are versioned *in place*, so a stale local copy is invisible from its file name — this one had read 2.31 GiB since it was first fetched while the Hub had moved on |
| `qwen38-flash-next-vision.hgn` | 0.84 GiB | vision tower (see below) |
| `tokenizer/` | ~21 MB | flat tokenizer dir; found beside the checkpoint, so no extra mount |
`HALOGEN_DOWNLOAD` fetches the whole repo into the bind mount on **first start
only** — the entrypoint acts only when the checkpoint is absent, so a restart
never re-downloads, and `huggingface_hub` resumes an interrupted pull. The
mount must be read-write for this. Roughly 127 GB; on Dankclaw it lands in
`/home/dank/models/halogen`.
**Refreshing the weights.** Delete the mount's contents (or the whole
directory, then recreate it **empty**) and start the unit: the entrypoint sees
the checkpoint absent and re-fetches the repo, resuming if interrupted. Two
traps. First, the bind mount's source directory must *exist* — podman refuses
the container with `Error: statfs /home/dank/models/halogen: no such file or
directory` and the unit loops on `Restart=on-failure`; `mkdir -p` the empty
directory and the fetch starts on the next start. Second, "refresh" only means
"fetch what the Hub has now": on 2026-09-16 the re-fetch brought newer sidecars
(`overlay.hgn` 2,477,677,120 → **2,572,466,560** bytes, `overlay-speed.hgn`
2,383,306,048 → **2,478,095,488**) while the checkpoint returned
byte-identical in size (124,068,083,904) — so an update can live entirely in
the sidecars, and equal file sizes are not proof of an unchanged repo.
`overlay-speed.hgn` (2.31 GiB on the Hub; 2.22 GiB before the 2026-09-16
refresh) comes along with the repo but is unused: it trades the `o_proj`
calibration for about 2 % of decode, and the quality sidecar is the default for
a reason.
## Deployed flags
Everything else is configured by environment variable — there is no config
file, and the flags are read once at startup. These are the departures from
upstream's defaults:
| flag | value | why |
|---|---|---|
| `HALOGEN_MODEL_ID` | `frank-dragoon` | fleet identity; no client rename |
| `HALOGEN_CTX` | `200192` | **set 2026-09-16**: exactly half the pool, so two conversations of the full window are always resident. Not a round 200,000 — pool/2 is. The model's native window is 262,144, so this is a real reduction in the per-conversation ceiling, and a client asking for more gets a 400 (never a silent truncation). The harness declares 200192 for `frank-dragoon` |
| `HALOGEN_KV_POOL_POSITIONS` | `400384` | **retuned 2026-09-16** (from 401,408): 2 x 200,192. Reserves ~11.0 GiB of KV, confirmed by the boot ledger as `400384 positions, ~27.2 GiB (16.2 GiB of it for everything but the pool, at max_tok 24576)` against a 33.5 GiB device budget — **no clamp**. Slots still oversubscribe the pool for any real request (admission reserves prompt + `max_tokens`), so a near-full pair that is both generating is pool-bound. `HALOGEN_KV_POOL_FIT` guards the boot: read the ledger, not this column |
| `HALOGEN_KV_SLOTS` | `2` | **lowered from 3 on 2026-09-16** — one slot per half of the pool, and two is the fleet's shape now that Dankclaw serves two endpoints. Each slot is 111.0 MiB of O(1) state, so this hands ~111 MiB back as well |
| `HALOGEN_MAX_TOK` | `24576` | **halved from the 32768 default on 2026-09-16** for frank-zealot, its co-tenant on the same host. The arena's variable term goes `8.40 x 2 → 8.40 x 1.5`, handing ~4.2 GiB back at roughly a 4-5 % prefill cost — the same lever at 16,384 costs 8-9 %. The vendor's own notes call 24,576 the largest arena that allocated with headroom to spare, so it stays inside their tested range. 65536 still cannot boot. It sizes a per-call PREFILL CHUNK, not the prompt ceiling: a 200k conversation is prefilled in arena-sized pieces |
| `HALOGEN_HOST_RESERVE_GIB` | `20` | **raised from 15 with the same retune**, because the arena shrank: `budget = 121.2 − 67.7 − 20 = 33.5 GiB` against a measured need of ~27.2, so the fit check kept ~6 GiB of margin and did not clamp. It is **not** an allocation: raising it cannot return memory to the host, only take the pool away (see the ceiling section below) |
| `HALOGEN_VISION_TOWER` | *unset* | **removed 2026-09-16.** The tower charges 1.97 GiB against the same device budget as the pool and the arena, and the fleet's image path moved to [[frank-zealot-qwen38-gsq-rco]] (llama.cpp with `--mmproj`). This server refuses image content parts, and `models.yml` declares `frank-dragoon` text-only to match |
| `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch |
| `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug |
Everything else is upstream's default, deliberately: prompt cache mode `2`
(two entries per conversation — end-of-system-prompt and end-of-history — which
already covers the shared-prefix case), `HALOGEN_CACHE_INPLACE=1` (the snapshot
stays ~115 MiB at any depth), the speculative drafter `1` (byte-identical at
temperature 0, and it stands down whenever a second conversation is
generating), the quality sidecar, `HALOGEN_ADMIT_CHUNK=0` (a non-zero value
gives up the byte-identity property), and the default `HALOGEN_MAX_TOKENS_CAP=65536`.
## Memory model
Three numbers explain the whole box:
- **~68 GiB pinned weights.** The checkpoint is mapped and registered with the
GPU in place, never copied. The kernel counts the locked pages as reclaimable
file cache, so `free`, `MemAvailable` and every dashboard built on them
**overstate free memory by the size of the model**. The server prints the
corrected figure at startup; believe that line, not Grafana.
- **~28 KiB per KV position** (measured: 262,144 positions = 6,656 MiB of KV,
~7.2 GiB with block scratch), plus the prefill arena. The 2026-09-16 ledger
reads the two together in the engine's own words: `400384 positions, ~27.2
GiB (16.2 GiB of it for everything but the pool, at max_tok 24576)` — so
~11.0 GiB of pool and 16.2 GiB of arena, working slack and (no longer) a
vision tower. The engine's device-side budget stops at roughly 47 GiB on a
128 GB machine *with the weights pinned* — and lower on a machine carrying
more besides this server, which Dankclaw is.
- **`HALOGEN_HOST_RESERVE_GIB`** is what is left for the page cache the lookup
table is read through. Too little and long prompts crawl with the disk busy
and the process in uninterruptible sleep — that is a cache problem, not a
memory problem, and the fix is a *smaller* pool, not a bigger one.
`HALOGEN_KV_POOL_FIT=1` (default) measures that budget at startup and lowers
the pool itself if the configured one will not fit, printing what it chose.
Read those lines on every restart; they are the ground truth for sizing.
**Measured ledger, seven restarts, read live 2026-09-12.** The engine's own
startup account, including three deliberate experiments:
| pool | max_tok | reserve | KV | arena | need | budget | verdict |
|---|---|---|---|---|---|---|---|
| 262144 (baseline) | 32768 | 24 | 7.2 | 20.6 | 29.8 | 29.5 | boots 0.3 over — prints **no** sizing lines |
| 524288 | 16384 | 24 | 14.4 | 12.2 | 28.6 | 29.5 | fits, 0.9 spare |
| 524288 | 32768 | 24 | 14.4 | 20.6 | 37.0 | 29.5 | clamped to 262144, deploy still green |
| 524288 | 65536 | 24 | — | 37.3 | 46.5 | 29.5 | **OOM — will not boot** |
| 524288 | 32768 | **15** | 14.4 | 20.6 | 37.0 | **38.5** | **deployed — both features** |
| 524288 | 32768 | 16 | 14.4 | 20.6 | 37.0 | 37.5 | fits, footprint unchanged — see the ceiling below |
| **400384** | **24576** | **20** | **11.0** | **16.4** | **27.2** | **33.5** | **deployed 2026-09-16 — ledger need 27.2, no clamp** |
| 401408 | 32768 | 15 | 11.0 | 20.6 | 33.4 | 38.5 | 2026-09-13 → 09-16, superseded by the retune |
### The sizing rule
One budget covers the pool, the arena **and** the vision tower together:
```
device budget = host RAM - resident weights - HALOGEN_HOST_RESERVE_GIB
= 121.2 - 67.7 - reserve
reserve 24 -> 29.5 GiB
reserve 15 -> 38.5 GiB
need = KV (28 KiB/position) + arena + vision (1.97 measured)
arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384)
```
The `121` the engine prints is rounded **down** from `MemTotal 127104060 kB =
121.2159 GiB`, so subtracting the displayed figures gives 29.3 rather than 29.5.
Use the unrounded value. Two of the three terms are fixed hardware and
checkpoint facts; `HALOGEN_HOST_RESERVE_GIB` is the only one you own — which
buys the *fit margin*, not the host's memory. It is a term in this inequality,
not an allocation, and the section at the end of this page shows the live test
that proves it.
Trust the engine's printed requirement over this model wherever they disagree:
a boot clamped at reserve 24 stated `524288 positions need ~35.0 GiB (plus 1.5
of margin)`, i.e. **budget >= 36.5**, against a modelled 37.0 here. Close
enough to plan with, and the authoritative number to cut the other way.
The arena term is **linear in `HALOGEN_MAX_TOK`** and predicts every measured
point to 0.1 GiB — including upstream's three published "on the device" figures
(524,288 → ~35, 786,432 → ~42, 1,048,576 at `max_tok=16384` → ~41), which are
**pool + arena totals at the default `max_tok`**, not pool alone. That is why
their largest pool "needs `HALOGEN_MAX_TOK=16384`": at 32768 a 1M pool would be
49.4. Reading those figures as pool-only is the trap that makes a doubling look
affordable when it is not.
At the baseline reserve of 24 the doubled pool leaves `29.5 - 1.97 - 14.4 =
13.1` for the arena, so `max_tok <= 18,130` and **16384 is that reserve's
ceiling, not a conservative setting.** The corner check agrees: at 32768 only
7.0 GiB is left for the KV, ~254k positions, and the pool drops back toward
262144. Cut the reserve to 15 and the same arithmetic gives `38.5 - 1.97 - 14.4
= 22.1` for the arena — `max_tok <= 35,700`, which is exactly why 32768 now
fits unmodified.
### What the reserve actually costs
The device budget is not the whole story. Memory position by configuration
(401,408 is the 2026-09-13 deployed row, read from its boot ledger):
| config | held | host left | contiguous 2 MiB blocks |
|---|---|---|---|
| 262144 / 32768, reserve 24 | 96.3 | 12.1 GiB | 2081 |
| 524288 / 16384, reserve 24 | 94.6 | 13.8 GiB | 3742 |
| 524288 / 32768, reserve **16** | **103.5** | **7.1 GiB** | 293 |
| **401408 / 32768, reserve 15** | **100.0** | **9.1 GiB** | **1396** |
| **400384 / 24576, reserve 20** | **95.4** | **14.7 GiB** | **475** |
At the doubled setting (2026-09-12) the server held 103.5 of 121 GiB and the
engine volunteered the stall note. It did not print it at the 401,408 trim, and
it prints it again after the 2026-09-16 retune. Three data points, and they do
not line up with a "free memory below X" reading: 310 blocks printed, 1396 did
not, 475 did. So treat the note as "the machine is tight" and read the
`host memory left` line rather than waiting to be told:
```
halogen: NOTE this server now holds most of the machine, and 310 contiguous
2 MiB blocks remain. It runs best on a host of its own. If other large
processes share this machine they will compete for what is left, and when it
runs out BOTH they and this server can stall for minutes at a time at full
CPU with no disk activity and no output, which is not a crash and needs no
restart.
```
That is the failure mode to watch, and it is the least diagnosable of the
three: not a silent clamp, not a boot refusal, but a minutes-long freeze that
looks like a hang, for the neighbours as much as for frank-dragoon. `free` is
useless for spotting it — `MemAvailable` still reads ~74 GiB because the
page cache counts as reclaimable. Watch the `host memory left for everything
else` line and the contiguous-block count at boot instead.
28 co-tenant containers (~4.3 GiB working set) stayed up through the change,
so it is survivable.
### The reserve cannot buy air — ceiling 16
The last row above is the experiment that proves it. Raising the reserve from
15 to 16 left `held` and `host left` **byte-for-byte identical** (103.5 / 7.1
GiB). `HALOGEN_HOST_RESERVE_GIB` feeds the fit-check budget and is *not* an
allocation; the actual footprint is `weights + KV + arena + vision`, none of
which the reserve touches. So the knob has exactly two regimes:
| reserve | budget | vs required 36.5 | effect |
|---|---|---|---|
| 15 | 38.5 | +2.0 | deployed |
| 16 | 37.5 | +1.0 | no change to footprint, thinner margin — pointless |
| 17 | 36.5 | +0.02 | knife edge, inside the noise of the 67.7 term |
| **18** | **35.5** | **−1.0** | **clamps the pool to 262144** |
| 24 | 29.5 | −7.0 | clamps the pool to 262144 |
The 36.5 requirement is the engine's own, read from a clamped boot:
`524288 positions need ~35.0 GiB (plus 1.5 of margin)`. Above 16 the only
thing a bigger reserve can do is take the doubling away.
**To actually free host memory, change the pool or the arena**, and the reserve
then becomes free to rise: `HALOGEN_KV_POOL_POSITIONS=262144` hands back
7.2 GiB from the doubled pool, `HALOGEN_MAX_TOK=16384` hands back 8.4. Both
are one-line edits that the fit check will confirm on the next boot. The
2026-09-13 trim to 401,408 was this remedy applied at half strength —
`HALOGEN_MAX_TOK` untouched, and on the host-left metric it bought +2.0 GiB
with a 4.5x improvement in contiguous blocks (310 → 1396), less than the
modelled 3.4 because the engine holds the freed KV as working slack.
### The `WARNING` line only appears when the fit check had to move
This one cost me a wrong conclusion, so it is worth stating plainly. When the
configured pool already fits, the engine reserves it **and prints nothing** —
the 09-11 baseline boot goes straight from the pre-flight line to
`checkpoint: mapped` with no `kv pool:` sizing block at all. The sizing block and
its `WARNING even 262144 positions ... host RAM cannot spare it` appear **only
after a clamp**.
So the absence of a warning in old logs is not evidence of headroom. Both
`262144/32768` rows above need 29.8 against a 29.5 budget — 0.27 over — and the
baseline has lived in that state since it was deployed without incident,
because the 24 GiB reserve is itself generous (the host reports 12–14 GiB free
at that point). The engine's budget is a conservative floor, not a hard wall.
### Two failure modes, and only one is dangerous
Raising `max_tok` past the ceiling does not shrink the pool — the arena alone
outgrows the budget, and the server cannot allocate at all:
```
kv pool: 524288 positions need ~51.7 GiB ... LOWERING THE POOL TO 262144 (~44.5 GiB).
kv pool: WARNING even 262144 positions (the context, the smallest pool that can
serve one request) needs ~44.5 GiB and host RAM cannot spare it.
dmalloc: FAILED requesting 0.250 GiB after 39.970 GiB in 566 allocations (out of memory)
```
This is the quiet one: the unit stays `active`, the container reports
`Up (starting)`, and `deploy.sh` sits in its health wait for the whole
`LLM_WAIT` rather than failing fast. It confirms upstream's warning that a
large `HALOGEN_MAX_TOK` means "the server will not start", not "it will be
slower".
Under-running the ceiling is far more benign and far easier to miss. At 524288
with the arena back at 32768, `HALOGEN_KV_POOL_FIT` lowered the pool and
`deploy.sh` still reported healthy in 12 seconds:
```
kv pool: 524288 positions need ~35.0 GiB (plus 1.5 of margin) and host
RAM cannot spare it. LOWERING THE POOL TO 262144 (~27.8 GiB).
```
A green health check does not mean the pool you asked for is the pool you got —
read the `kv pool:` lines after every restart.
The 47.7 GiB n-gram lookup table is never held in RAM; it is read through
the file cache. The 2.0 GiB the firmware carves for the iGPU appears in no
`/proc/meminfo` figure at all.
**Co-tenancy.** `HALOGEN_FLASH_PIN_TRUNK=1` (default) is what the published
decode numbers were measured with, and it means this container holds most of
the host. If a co-tenant then takes the remainder, both it and the engine can
stall for minutes at a time at full CPU with no disk activity and no output.
That is documented behaviour, not a crash, and it needs no restart.
`HALOGEN_FLASH_PIN_TRUNK=0` gives the memory back and costs several times the
decode speed — a last resort, not a tuning option.
## Client contract
The differences from llama.cpp that bite are all in the request, not the
deployment:
- **No server-side sampler defaults.** `temperature` absent or `0` is
**greedy** decode; above 0 the request samples on the same drafter. Supported
fields: `temperature`, `top_p`, `top_k`, `min_p`, `seed`,
`presence_penalty`, `frequency_penalty`, `logit_bias`, `logprobs`. Refused
with a 400: `top_logprobs`, `logprobs` with `stream: true`, `n > 1`, and any
value outside its defined range (values are refused, never clamped).
The fleet runs greedy for clients that stay silent — deterministic, and
byte-identical with the drafter.
- **The token budget covers thinking.** The default is **8192**, and a budget
that runs out mid-thought returns `finish_reason: "length"` with an **empty
`content`** and the partial reasoning in `reasoning_content` — which most
clients do not display. Above the cap (`HALOGEN_MAX_TOKENS_CAP`, 65536) a
request is a 400 rather than a silent truncation. `max_tokens`,
`max_completion_tokens` and `max_output_tokens` all work and must agree if
more than one is sent.
- **A budget above (window − compaction-threshold) is a guaranteed 400 zone.**
The fit check (`prompt + max_tokens <= HALOGEN_CTX`) refuses, and the harness
only compacts at its own threshold — with OMP at 70 % of a 200,192 window
(140,134), any prompt in (136,192, 140,134] with a 64,000 budget overflows
the window while sitting below the compaction trigger. That collided live on
2026-09-17 (prompt 138,898 + 64,000 = 202,898 > 200,192). The fix is
harness-side: keep the declared `maxTokens` at or under
`window − threshold` (57,344 chosen) so no pre-compaction request can miss
the fit check. Do not fix it by raising `HALOGEN_CTX` past pool/2 — that
breaks the two-resident-conversations property.
- **`reasoning_effort`** is accepted at the top level and inside
`chat_template_kwargs`: `minimal`, `low`, `medium`, `high`, `xhigh`. The
model's default is `xhigh`. `minimal` aliases the template's `low` and still
reasons; `enable_thinking: false` turns thinking off; `preserve_thinking` is
honoured. `preserve_reasoning` is a **400** — it is the llama.cpp-era key.
- `/v1/models`, `/health` and completions all report the served id, and
`reasoning_content` carries the reasoning trace.
**Pool admission is the one arithmetic trap.** A request reserves its prompt
**plus** `max_tokens` positions when it is admitted; if the pool cannot hold it
the request waits in arrival order, and after the queue timeout (3600 s) it
gets a `503 engine_busy`. A prompt large enough that prompt + budget exceeds
the pool can therefore never be admitted at all. The 2026-09-16 shape is a
400,384-position pool with `HALOGEN_CTX` 200,192 and two slots: two
conversations at the ceiling fit the pool exactly, so a pair that is both
near-full and both generating is pool-bound — the second waits for the first to
release its region. The pool binds **how many** conversations stay resident,
not how long one may be; the older 524,288 / 401,408 numbers live in the ledger
tables above.
## Vision
**None here since 2026-09-16.** `HALOGEN_VISION_TOWER` is deliberately unset:
the tower is a 1.97 GiB charge against the same device budget as the pool and
the arena, and the fleet's image path moved to [[frank-zealot-qwen38-gsq-rco]],
which loads the BF16 projector alongside its own weights. This server refuses
image content parts and says why, rather than ignoring them:
```json
{"error": {"message": "this server does not accept images: the engine was
started without a vision tower (HALOGEN_VISION_TOWER)", "type":
"invalid_request_error"}}
```
The llama.cpp side handles images the same way this engine used to: a `data:`
URL or bare base64, about 2,040 tokens of context per image, and `http(s)` URLs
refused deliberately (fetching one would make the server issue outbound
requests to wherever a client pointed it). Audio and video have no path
anywhere in the fleet — that left with [[frank-npu]].
## Deploy and operate
```bash
./scripts/deploy.sh dankclaw --service frank-dragoon
```
The handler retires the ik-era **root** unit first (stop, disable, move
`/etc/systemd/system/frank-dragoon.service` into
`~/.AGENTTRASH/20260916-halogen-restore/host-units` — never `rm`; this host's
`disable` may delete the file outright, which the handler now reports), stops
the Quadlet's own unit if it is running, then pulls
`ghcr.io/peonist-ai/halogen-flash-server:0.5.8`, re-tags it to the
`localhost/` alias the unit references, and **waits for port 7770 to be free**
before installing and starting anything.
That last gate is not paranoia. On the first cutover attempt the ik server
logged "Deactivated successfully", and pasta — rootless podman's userspace
network stack, which binds the published port at container CREATE — still
reported `Address already in use` a second later, exiting the unit with code
126. Only `Restart=on-failure` recovered it, minutes later. A deploy that
starts a doomed container has not deployed anything, so `wait_port_free` polls
every socket on the port (listener, accepted, TIME_WAIT) and fails loudly with
the socket table if it never clears.
Boot persistence comes from `[Install] WantedBy=default.target` in the unit
file plus linger on the deploy user — generated units **cannot** be
`systemctl --user enable`d on this host.
```bash
systemctl --user status frank-dragoon.service # the generated unit
podman logs frank-dragoon | head -40 # version banner + memory ledger
curl -s http://127.0.0.1:7770/health # 200 only after the engine PONGs
curl -s http://127.0.0.1:7770/v1/models
```
`/health` asks the **engine** for a PONG before answering, so a 200 means
"ready to serve", not merely "process alive". During the first-start download
it stays unhealthy by design.
## Troubleshooting
| symptom | cause |
|---|---|
| `dmalloc: FAILED requesting … (out of memory)` then `HIP … out of memory` | the KV pool did not fit. `HALOGEN_KV_SLOTS` will not fix it (slots share the pool); lower `HALOGEN_KV_POOL_POSITIONS` to 262144, then `HALOGEN_MAX_TOK` to 16384 (~9 % prefill) |
| starts, answers short prompts, collapses to a few tok/s on a long one, disk busy, process in uninterruptible sleep | short of **file cache**, not memory — lower the pool and/or raise `HALOGEN_HOST_RESERVE_GIB` |
| long replies that look cut off | read `finish_reason`: `stop` means you have the whole answer, `length` means you ran out of budget (thinking counts) |
| a request that never returns, then `503 engine_busy` | the pool could not reserve prompt + `max_tokens`; see the admission trap above |
| both the engine and a co-tenant stall for minutes at full CPU | the pinned-trunk trade; see Memory model |
| `free`, `MemAvailable`, Grafana all wrong by ~68 GiB | expected — locked weights are counted as reclaimable |
`HALOGEN_VERBOSE=1` ships in the unit, so the fullest startup account is
already in the log when something goes wrong. `HALOGEN_DMALLOC_LOG` is
deliberately **not** set: it logs every allocation over 64 MB, which is
steady-state noise on a shared host — raise it only for a live OOM report.
The lines worth grepping first:
```bash
podman logs frank-dragoon 2>&1 | grep -E '^(dmalloc|kv pool):'
```
Retrieval: `podman logs frank-dragoon` covers the **current** instance only —
the Quadlet runs `--replace --rm`, so everything before the last recreate is
gone from the container log. `journalctl --user -u frank-dragoon.service` is
the durable record. Both are rootless: `sudo podman logs frank-dragoon` looks
in root's storage and answers "no such container".
## Rollback
The predecessor is intact: the ik payload stays at
`/home/dank/models/mradermacher-Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`, the
ik build at `/home/dank/GitPackages/ik_llama.cpp/build`, and its unit, env,
serve script and the Qwen-Sharp template are archived in the repo under
`scripts/.AGENTTRASH/*.ik-llama.cpp` and `*.ik-llama-qwen-sharp`. Rolling back
is a `git revert` of this change set plus a deploy — **but the Quadlet must be
removed from `~/.config/containers/systemd/` first**, or the restarted root
unit and the still-installed container both claim port 7770. The port gate in
the handler now covers exactly that race. See [[frank-dragoon-ik-uncensored]].
The `amd_iommu=off` kernel argument that came with this change is separate and
reverts on its own — see [[dankclaw-iommu-translation-off]].
## Citations
- [peonist-ai/halogen-flash-server](https://github.com/peonist-ai/halogen-flash-server) — README, `docs/FLAGS.md`, `deploy/entrypoint.sh`, `docker-compose.yml`, CHANGELOG (v0.5.8)
- [peonist-ai/halogen-qwen3.8-flash-next](https://huggingface.co/peonist-ai/halogen-qwen3.8-flash-next) — weights and file inventory
- [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) — base model
serverAppConfigs-docs/LLM/index.md
+80 -638
View File
@@ -1,647 +1,89 @@
# LLM Fleet — Knowledge Base
# LLM fleet — index
**Dankserver runs the radiance engine (StillDeadcode) on frank-archon**
(digest-pinned `stilldeadcode/radiance` image, engine as entrypoint,
driving `podman run` directly, root systemd unit, :7770):
the **publisher's** 4-bit-codebook container of the official
`Qwen/Qwen3.8-Flash-Next` (`qwen3.8-next-flash-fp8-iq4r-moe.rad`, 116,322 MiB,
sha256-verified at fetch — w4nl codebook experts + int8 trunk, PLE n-gram,
MTP head and vision tower in-container), on BOTH R9700s
TP=2 (`--tp-wire wht6`), experts in a 44 GiB/rank pinned host pool + VRAM cache
(explicit `--vram-weights-mib 25600 --vram-kv-mib 5120` = 30,720 of the
30,969 MiB the engine's own budget table makes claimable), the 47.7 GiB PLE
table pinned to host RAM, **fp8 KV at 3 × 200,000**, MTP spec decode depth 2,
froggeric v22.5 template. Rollback: revert the swap commit and re-download the
in-house Swift1.5 w4dn8 `.rad` byte-exact (purged from disk 2026-10-17; Hub
`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`); tags
frank-archon-freetoken-pr587-20261005 and frank-archon-vllm-shapeG-20261003
(AGENTS.md Runbook Index; `find /home/dank/models -name '*.rad'` on
Dankserver 2026-10-18 lists only the iq4r file),
and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2
(frank-dragoon, port 7770: ghcr.io/peonist-ai/halogen-flash-server @ pinned
digest serving the NATIVE v2 conversion of the abliterated
Qwen3.8-Flash-Next, a ~62 GiB v2 .hgn that loads in no other engine,
MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots over ONE
401,408-position KV pool — 2 × ctx, raised and DEPLOYED 2026-10-18 — at ctx
200704, vision ON)** and
**llama.cpp on the PrismML fork (frank-zealot, port 7771: prism-ml
Ternary-Bonsai-2-9B at **PQ2_0** — 9B qwen35 hybrid (8 full-attention + 24
Gated DeltaNet layers + the blk.32 MTP block), TWO slots over ONE
524,288-position pool (2 × the 262,144 window), MTP present in the file but
OFF by measurement, **exact f16 K + f16 V** — 32,768 B/token = **16,384 MiB**
of KV (three 262,144 windows at exact KV do not fit the RTX 3090), vision via
the publisher's own Q8_0 mmproj, on the RTX 3090 — and the custom
tensor type 141 + `prism.hadamard.*` metadata mean no other loader can open the
file)**.
The three live endpoint pages are the only pages that carry a shape. Everything
else here is a technique, a piece of hardware, or a trial that taught a law — and
each of those owns exactly that one thing. The incidents behind the laws are in
[[log]].
> 2026-08-22: the former `vllmScripts` repository is merged into the
> serverAppConfigs repository. This knowledge base moved from the old repo's
> `serverAppConfigs-docs/` root to `serverAppConfigs-docs/LLM/`. The fleet scripts and units now live in the
> repo's `scripts/` directory.
## Live endpoints
## Deployment Topology
| Page | Owns |
|---|---|
| [[frank-archon]] | The one frank-archon page: the live radiance shape and harness contract (read off scripts/frank-archon.env), the Eras… |
| [[frank-dragoon]] | The one runbook for frank-dragoon, the Strix Halo iGPU endpoint (192.168.0.143:7770): current shape from the env,… |
| [[frank-zealot]] | The one frank-zealot reference: the live shape on Dankclaw's RTX 3090 (:7771) exactly as scripts/frank-zealot.env… |
```mermaid
graph LR
subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
A[2x Radeon AI PRO R9700
gfx1201, 32 GiB each]
B2[radiance StillDeadcode, digest-pinned image
podman run, frank-archon, port 7770]
C[Publisher qwen3.8-next-flash-fp8-iq4r-moe .rad
2x R9700 TP=2 wht6, experts 44 GiB/rank pinned pool + VRAM,
PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 2,
3 x 200000, froggeric v22.5]
end
## Retired endpoints
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
E2[RTX 3090 (PCIe x4 dock) +
gfx1151 iGPU 114 GiB GTT]
E[flash-server container gfx1151, digest-pinned image
frank-dragoon, port 7770]
F[flash-server v2 native abliterated .hgn + ngram table + vision tower
ONE 401408-position KV pool at ctx 200704 = 2 slots x ctx, MTP always-on, image input]
G[llama.cpp PrismML fork CUDA build
frank-zealot, port 7771]
H[Ternary-Bonsai-2-9B PQ2_0 GGUF + Q8_0 mmproj
2 slots over ONE 524288-position pool at ctx 262144, exact f16 K + f16 V, MTP off, mmproj vision]
E2 --> E --> F
E2 --> G --> H
end
| Page | Owns |
|---|---|
| [[frank-medium-batch-ubatch]] | llama-bench sweep of --batch-size against --ubatch-size on frank-medium (2x Radeon AI PRO R9700, layer split) at the… |
| [[frank-medium-llamacpp-dflash2]] | The former ACTIVE frank-medium deployment since 2026-09-16 (model swapped 2026-09-19) — mainline llama.cpp on… |
| [[frank-medium-model-swap]] | How to swap the checkpoint behind the frank-medium endpoint (Dankserver :7770, vllm-radiance TP=2) — the streamlining… |
| [[frank-medium-radiance-mxfp4]] | ACTIVE frank-medium runbook — vllm-radiance serving hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 (ParoQuant MXFP6 W4A8,… |
| [[frank-medium-rocmfpx]] | charlie12345/ROCmFPX llama.cpp serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF as frank-medium on Dankserver… |
| [[frank-npu]] | The retired FastFlowLM endpoint on Dankclaw's XDNA2 NPU — what it served, why it died on 2026-09-11 with the… |
| [[frank-redo-2026-08-17]] | This page records the full redo of the inference fleet on 2026-08-17. |
| [[ikllama-ornith-dankserver]] | ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-zealot on the RTX 3090. Q8_0 model, KV K default f16 / V… |
| [[kingjones-q8agent-rocmfpx]] | **Status (historical, 2026-09-13):** was to deploy on Dankclaw. |
| [[qwen3.8-spec-matrix]] | This page records the speculative-decoding benchmark history and the |
CLI[LLM Clients / API] -->|HTTP :7770| B2
CLI -->|HTTP :7770| E
CLI -->|HTTP :7771| G
```
## Pipeline Parallelism (series)
## Servers
| Page | Owns |
|---|---|
| [[Ethernet Bandwidth Math]] | Bytes-per-token math for pipeline vs tensor parallelism, feasibility tables for 1/2.5/10 GbE, and real multi-node… |
| [[Expected Performance]] | Estimated tokens per second for a 25-40B model split across two nodes over 1/2.5/10 GbE, with real benchmark anchors. |
| [[llama.cpp RPC]] | llama.cpp distributed inference via ggml-rpc-server and the --rpc flag: architecture, CLI, versions, limitations, and… |
| [[Mixed-Vendor Experiments]] | Real-world NVIDIA + AMD Strix Halo multi-node experiments: what works (llama.cpp RPC, Vulkan split), what breaks… |
| [[Qwen3.6 27B Spec Methods]] | Qwen3.6-27B Q8 GGUF variants and llama.cpp speculative decoding (draft-mtp, draft-dflash, ngram-mod): flags, RPC… |
| [[Qwen3.8 PP Deployment]] | Split Qwen3.8-27B across the Strix Halo iGPU (head) and the RTX |
| [[Sample Script Set]] | An educational two-node llama.cpp RPC split of Qwen3.6-27B-Q8_K_P over ethernet. For reading only. Do not run or… |
| [[vLLM Pipeline Parallelism]] | vLLM multi-node pipeline parallelism: flags, Ray/mp backends, interconnect requirements, PP bubble, and PP +… |
| Server | GPU | VRAM / Memory | OS |
| --------------------------------------- | ----------------------------- | -------------------- | ------------ |
| [[Dankserver System Specs\|Dankserver]] | 2x AMD Radeon AI PRO R9700 | 2x 32 GB VRAM, 246 GB RAM | Ubuntu 26.04 |
| [[Dankclaw System Specs\|Dankclaw]] | AMD Radeon 8060S (Strix Halo) | gfx1151 unified memory | Ubuntu 26.04 |
## Techniques, hardware, trials
## Active Deployment
- [[frank-dragoon-flash-server-v2-native]] — **ACTIVE since 2026-10-17**
(root systemd unit + podman, port 7770): the digest-pinned
`ghcr.io/peonist-ai/halogen-flash-server` 0.16.2 engine serving the
**native v2 conversion** of the abliterated Qwen3.8-Flash-Next
(`qwen38-flash-next-v2-abliterated.hgn`, 66,687,678,432 B, Hub LFS
sha256-verified at fetch) with the REQUIRED n-gram lookup table (51.2 GB,
@peonist 5cc17cea) and the vision tower (897.9 MB, @e053f488)
**hardlinked** beside it — the card publishes only the main file, and the
engine's own tools certified the stock companions (`niah` 18/18, `ppl`
3.968 with mean KL 0.155 against the Swift1.5 logits on the same corpus).
`HALOGEN_CTX=200704` over `HALOGEN_KV_POOL_POSITIONS=401408` (**2 × ctx**):
**one pool behind two slots**, sized so TWO maximum-length requests coexist —
raised from 200,704 (5.5 GiB) on 2026-10-18 for frank-zealot parity on total
in-flight context and **deployed the same evening**: the boot ledger reserves
`401408 positions (about 11.0 GiB)` with `79.6 GiB in all`
(`scripts/frank-dragoon.env:70-75`). It bought CONCURRENCY, not a longer
window: the per-request ceiling stays 200,704, because 262,144 is the
native wall and the engine refuses more without `HALOGEN_ROPE_YARN`.
MTP head always on, sampler 1.0/0.95/20, coupled cap/timeout
65536/7200, prompt cache ON, vision ON. Measured against the Swift1.5
boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18,
PPL +2.3 %, thinking tokens ×1.9 (the finetune's compression is gone).
Rollback: revert the swap commit; the swift15 payload and its fetch script
stay on disk
- [[frank-dragoon-flash-server-v2-swift15]] — **RETIRED 2026-10-06 → 10-17
(rollback target: payload + fetch script on disk)**: the Swift1.5-finetuned
v2 `.hgn` (compacted r2, 66,687,678,464 B) at TWO slots at ctx 262,144 over
a 524,288-position pool; owns the 400-zone law (the harness chars/4
under-count against the engine ceiling), the MTP-always-on law, and the
sampler / prompt-cache / vision contract the native era carries verbatim
- [[frank-dragoon-halogen27b-dflash2]] — **RETIRED 2026-10-05 → 10-06
(rollback target; payload + fetch script + image on disk)**: the
digest-pinned `ghcr.io/peonist-ai/halogen` 0.1.4 engine serving
`peonist-ai/halogen-qwen3.8-27b` (`qwen3.8-27b-p1w4d-d2.hgn`, 33.4 GiB)
— DFlash2 drafter, ONE 200,700 slot, text-only (the engine ignored the
vision encoder), censored base. Owns the measured lessons: the prompt
cache pin (`HALOGEN_CACHE_MB=16384` — auto was OFF at the 36.9 GB
two-entry floor; a 17.3k cold prefill of 27.0 s resumed in 1.7 s warm),
the FA tile rejection (`HALOGEN_FA_TQ=256`: prefill −9.5 %, decode
±0.3 %), and deep decode at 3.3-3.8 t/s@80k being structural. The
foreground-`podman run` ledger capture and the deep-role rerouting
(`slow`/`advisor` → frank-archon) carried forward. Superseded by the
MODEL swap: the abliterated v2 .hgn loads only in flash-server, so the
engine moved with the model
- [[frank-zealot-llama-prism-bonsai9-mtp]] — **ACTIVE since 2026-10-18**
(root systemd unit, port 7771): the **PrismML fork** build of llama.cpp
(vendor **10777**, commit `e3de71522` — the only loader for the payload's
custom PQ2_0 tensor type + `prism.hadamard.*` transform, and since this swap
also the minimum build that pairs MTP with a quantized V cache) serving
prism-ml/Ternary-Bonsai-2-9B-gguf **PQ2_0 with the MTP head IN the file** on
the RTX 3090 — **TWO 262,144-token conversations**
(`--ctx-size 524288 --parallel 2`, every slot at the window the model was
trained at), **unquantized f16 K + f16 V** (32,768 B/token over the 8 full
layers = **16,384 MiB** exact; the POOL halved because that rate over the
OLD 786,432 pool bills 24,576 MiB against the 24,123 the driver leaves
addressable, and the old "26,112 MiB" note was wrong — it doubled the
q8_0 figure (17,408 B/token = 13,056 MiB) *including* its 1/32 scale
overhead, so f16/q8_0 is **1.882×**, not 2×), the q8_0-era boot settled
**17,876 used / 6,249 free**, vision via the publisher's **Q8_0** mmproj
(Q8_0 is the publisher's requirement for concurrent image requests, and this
runs two slots), **MTP OFF by measurement** — the draft context bills its own
f16 KV over the WHOLE pool (4,096 B/position × 524,288 = 2,048 MiB) and at
`UBATCH_SIZE=2048` that squeeze halves decode (167.2 → 109.2 t/s);
`SPEC_TYPE=draft-mtp` + `UBATCH_SIZE=1024` is the documented 213.5 t/s arm,
declined for +17 % prefill and 6.2 GiB of headroom — and exact KV leaves an
f16-draft arm no room at all (a quantized draft, `--spec-draft-type-* q8_0`,
is the unmeasured door).
Measured at the q8_0-era shape: 167.2 t/s single-stream decode, 2,877 t/s
prefill at 106,418, 280.2 t/s aggregate across three streams. `--metrics`
rides (`METRICS=1`):
llama.cpp answers 501 on `/metrics` without it, which is why Prometheus
`up{job="frank-zealot"}` read 0 for its whole retained history until
2026-10-18. `contextWindow: 262144` is the **widest in the fleet**, so this
is the compaction hub. The `--presence-penalty` opt-out law (`${VAR=}`, not
`${VAR:=}`) and the `TimeoutStopSec=180` stop-drain law were both earned here.
- [[frank-zealot-llama-prism-bonsai2-pq20]] — **RETIRED 2026-10-17 → 10-18**
(payload + fork build 10754 tree on disk as rollback): the first PrismML-fork
era — prism-ml/Ternary-Bonsai-2-**27B**-gguf **PQ2_0**, TWO 200,704-token
conversations (`--ctx-size 401408 --parallel 2`), q8_0 K + q8_0 V
(34,816 B/token over the 16 full layers = 13,328 MiB; f16 needs 25,088 MiB and
dies), settled 22,664 used / 1,461 free, MTP **impossible** (that export
carries no head), 76.6 t/s decode, 691.7 t/s prefill at 166k, 109.4 t/s
aggregate at two streams. Owns the 34 KiB/token KV law, the f16 OOM wall, the
two-boot VRAM pricing, and the record of the 2026-10-18 bench window
(production unit stopped 03:38, hand-booted 9B candidate answering the port)
that closed with this swap.
- [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — **RETIRED 2026-10-07 →
10-17** (payload + master CUDA build on disk as rollback): llama.cpp master
(CUDA sm_86) serving HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive
**Q6_K** on the RTX 3090 — TWO 200,704-token conversations
(`--ctx-size 401408 --parallel 2`), exact f16 K + f16 V, vision via the
publisher's BF16 mmproj. MTP is OFF because the file has no MTP tensors at
all. Priced from the MiMo Q6_K boots (same qwen35 geometry, same card, same
shape). Supersedes the Gemma4 era (below).
- [[frank-zealot-llama-gemma4-q4km-200704]] — **RETIRED 2026-10-07** (same
day; superseded by qwen35-9b-q6k; payload on disk as rollback): llama.cpp
master (CUDA sm_86) serving HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-
HauhauCS-Balanced-MTP **Q4_K_M** on the RTX 3090 — ONE 200,704-token
conversation, f16 K + q8_0 V, vision via the publisher's mmproj. MTP was
OFF (the gemma4-assistant drafter costs ~1.9 GiB: 23,806 used/319 free with
it, 21,890/2,235 without).
- [[frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688]] — **RETIRED
2026-10-07** (superseded by gemma4-q4km; payload on disk as rollback);
ACTIVE 2026-10-05 (evening) → 2026-10-07; UNCECERSED RCO-IQ3_XXS with
MTP back on since
2026-10-06 evening; RENAMED from frank-small and reshaped to 2 ×
114,688 / ubatch ladder 512 → 1024 → **2048** late 2026-10-06** (root
systemd unit, port
7771): llama.cpp master
(CUDA sm_86) serving ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-
GGUF **RCO-IQ3_XXS-mtp** (27B; the MTP head in the file is DRAFTED —
draft-mtp depth 3, ngram skipped on the recurrent-rollback law) on the
RTX 3090, TWO 114,688-token conversations at Frank's 112k cap (pool
229376 / parallel 2 — cut from 131,072 to fund the ubatch rungs;
**1,005 MiB free at boot, 763 settled after a vision probe**) with
q8_0 K + q8_0 V (FP8 class, 34 KiB/token vs 64 at f16) and
vision through the publisher's own mmproj-BF16.gguf. Prefill 1,209
t/s @ 3,760; decode ~52–58 t/s recipe (ubatch-invariant; acceptance
noise 34–54 %).
RCO-IQ3_XXS tier is a per-tensor allocation mix of stock types; the
payload is the UNCECERSED Swift 1.5 (23/100 refusals vs Swift 1.5's
98/100). Replaces BOTH same-day payloads: the live MiMo Q6_K shape
(below) and the Ornith-1.5-9B Q8_0 swap that was committed
(77decb5..380aafa) but never deployed
- [[frank-zealot-llama-mimo-q6k-2x200704]] — **RETIRED 2026-10-05 (evening;
rollback target; payload on disk)**: llama.cpp master (CUDA sm_86)
serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K**, TWO slots of
200,704 with FULLY UNQUANTIZED KV (f16 K + f16 V) and the mmproj for
vision — 94.4 tok/s decode (the rung costs a measured -10 % vs Q5_K_M:
weight-bandwidth-bound), batch 4096 / ubatch 2048 since 2026-10-05
(+4.6 % prefill, measured), 2,279 MiB of the card left free. This was
the LIVE payload when Swift cut over. The ubatch A/B table and the
probe-size trap (>=256 px images or the model says "video") live inside
- [[frank-zealot-llama-mimo-q5km]] — **RETIRED 2026-10-02 → 10-03 (rollback
target; payload on disk; owns the measured boot-pair table)**: the same
runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower —
104.8 tok/s, 3,571 MiB true free
- [[frank-archon-radiance-qwen38-iq4r]] — **ACTIVE since 2026-10-15**:
the engine author's published container of the OFFICIAL (base-alignment)
Qwen3.8-Flash-Next — `qwen3.8-next-flash-fp8-iq4r-moe.rad`, 116,322 MiB,
Hub-sha256-verified at fetch: 4-bit codes into a 16-level non-uniform
codebook after a Walsh–Hadamard rotation (GPTQ + 3 coordinate-descent
sweeps, 10M-token calibration), ten bf16-protected experts per layer,
int8 trunk/lm_head, fp8 HC + PLE, MTP head, bf16 vision tower; card KL
0.0789 all-tokens / top-1 92.85 % vs bf16. **MODEL_FILE is the only line
that moved** — the 2026-10-13 shape (25600/5120 pair, 44928 pool, fp8 KV
3 × 200,000, MTP 2, wht6, MNBT 2048, froggeric v22.5) carries verbatim.
Rollback: revert the swap commit (w4dn8 `.rad` stays on disk). The
fp8-native `.rad` and the raw Swift1.5 BF16 source were DELETED from
dankserver 2026-10-15 (Frank's one-time rm, volume for this payload;
rebuild via the fetch + build scripts named in the fp8-native runbook)
- [[frank-archon-radiance-swift15-w4dn8]] — **RETIRED 2026-10-08 → 10-15
(superseded by the publisher iq4r swap; the `.rad` was PURGED from disk
2026-10-17 — rollback = revert + byte-exact re-download of
`frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700`)**:
Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 `.rad` on radiance
1.0.13 (recipe + 10M-token calibration + KLD gate vs the BF16 container);
the FP8-native payload was the second rollback until it too was deleted
2026-10-15. **2026-10-13: fp8 KV, and the
pair re-cut twice** — `--kv-cache-dtype fp8` first at
`--max-num-seqs 4`/6912 MiB pool (843,566 tokens, slab 22208), then **3 slots
with the pool cut to the contract (5120 MiB, 608,027 tokens) and the slab
raised 22208 → 24000 → 25600**, the last 1,600 MiB claimed out of the card's
settling slack: the engine's ceiling is **30,969 MiB**, not the 29,120 my
1 GiB reserve invented. fp8 is what funds the slots — the pool must back
`seqs × window`, and at bf16 four slots cost 11,679 MiB (slab 17,441, −33 %).
Measured on the slab: A-cold 43.5k prefill 22208 → 3,397, 24000 → 3,290,
**25600 → 3,584 t/s**; 3-way 197k aggregate 3,051 → **3,589 t/s**. A shipped
prediction that the slab was a threshold below 26 GiB died on its own
measurement — it is a slope, and 26,071 (4,379 t/s) is the cliff. The bf16
arm (2 × 200,000, 462,752 tokens, prefill −29 %, acceptance 0.661 → 0.740)
and the 4-slot fp8 shape are the documented rollbacks.
**2026-10-13 also: the pinned-ledger preflight now raises `ttm.pages_limit`
itself** — the 2026-10-12 power failure
reset the knob to its default (125,988 MiB) and the endpoint refused to boot
6,919 times in ~20 h; `scripts/ttm-preflight.sh` is the repair.
- [[frank-archon-radiance-qwen38-native-fp8]] — **RETIRED 2026-10-08 (the
w4dn8 swap replaced it; its `.rad` was DELETED from disk 2026-10-15 —
rebuild = `scripts/download-frank-archon.sh` + `scripts/build-frank-archon-rad.sh`,
`scripts/frank-archon.env:59-62`)**
(root systemd unit, port 7770): the StillDeadcode **radiance** engine
(C++/HIP, libr4d gfx1201 kernels, digest-pinned 287 MiB image, engine as
the image entrypoint) serving the OFFICIAL `Qwen/Qwen3.8-Flash-Next-FP8`
checkpoint at NATIVE precision — converted no-recipe to
`qwen4exp-fp8-native.rad` (~161 GiB) after dropping `quantization_config`
in a staging dir (the plugin descriptor-selection law: nothing claims
`(qwen4exp, "fp8")`, the shipped plugin claims `(qwen4exp, "")`;
per-weight encodings are data, not the descriptor) — on BOTH R9700s
**TP=2** (`--tp-wire wht6` fused all-reduce), experts in a 40 GiB/rank
pinned host pool + VRAM cache (ratio 0.82 = the measured 99 %-hit point),
the 47.7 GiB PLE n-gram table PINNED to host RAM (`--ngram-placement
ram`), fp8 KV 7.32 KiB/token, MTP spec decode depth 3, 2 × 200,000 first
boot, froggeric v22.5 template, vision + tools + `reasoning_content` auto,
`/metrics` scraped. Rollback: git tag
`frank-archon-freetoken-pr587-20261005` (FreeToken image + orcarouter
payload on disk); older rollback `frank-archon-vllm-shapeG-20261003`
- [[frank-archon-freetoken-pr587]] — **RETIRED 2026-10-05 (same day;
rollback target)**: FreeToken at PR #587 head `77a7201` (branch
rocm714_torch214) built on the host into the local image
`frank-archon-freetoken:pr587-77a7201` (Dockerfile: digest-pinned
rocm/pytorch rocm7.14 torch-2.11.0 base, libhiprtc dev-symlink law,
curated --no-deps install per docs/install_amd.md), serving the SAME
orcarouter FP8 payload on ONE R9700 — experts in pinned host RAM
(~120 GiB + 47.7 GiB PLE), GPU LRU at 0.10, KV pool 299,968 tokens
(multiple-of-64 law), 2 slots, parsers AUTO (thinking →
`reasoning_content`). Measured: boot ~2.3 min, decode 17.1 tok/s
single-stream, vision "Red" probe passes, native tool_calls; TP=2 does
not boot on ROCm (pynccl JIT) — second card idle. Superseded by the
radiance swap for TP=2. Disk state 2026-10-18: the image is still in
dankserver's rootless storage (`localhost/frank-primary-freetoken:pr587-77a7201`,
29.7 GB — the pre-rename name); the orcarouter FP8 payload is NO LONGER on
disk (no `*orcarouter*` path under /home/dank, `find` 2026-10-18)
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **RETIRED 2026-10-05
(rollback target; the vLLM trio + env lore live in the tag)**: the
tcclaviger RDNA4
vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both
R9700s, TP=2 with automatic expert offload — `--max-model-len 200704` x 3
slots, fp8 KV, pool 3,150,000,000 B/rank (measured 398,658 tokens = 1.99x),
MNBT 1024, gather 32x64, no spec decode, prefetch off, plus
`--expert-precise-memory` (arena 17.22 GiB / 7,522 slots), GPU_MAX_HW_QUEUES=2
and a mounted compile cache. Why `--max-num-seqs`
must stay 3 (SEQS 2 cannot compile this model), why the arena's levers are
`max_model_len` and the overhead term rather than the KV pool, and the measured envelope
- [[frank-archon-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
failed GLM experiment; frank-archon runs Qwen FP8 again, see the intro):
the tcclaviger RDNA4 vLLM fork
(digest-pinned `tcclaviger/vllm@sha256:da4029666…`, plain `podman run`,
no shim) serving Intel/GLM-5.3-Flash-W4A16-AutoRound — AutoRound symmetric
int4 group 128, ~170 GiB, 34 shards + 2 extras, 113,074 tensors — on BOTH
R9700s, TP=2, with the MoE experts offloaded to pinned host RAM (12,096
experts/rank, 73.64 GiB) behind a 14.51 GiB / 2,383-slot VRAM LRU arena; the
MoE kernel is libr4d's grouped W4A16 GEMM (`R4dW4a16MoEMethod`, reached
because the checkpoint's `quant_method: auto-round` is overridden to `inc` —
a compressed-tensors W4A16 checkpoint of the same model has NO offload recipe
and refuses to serve). 131,072 ctx x 2 seqs, bf16 KV, util 0.98, prefix
caching, vision and `glm47` tool/reasoning parsers, MTP OFF and prefetch OFF
by measurement. **Measured decode 9.9-10.0 tok/s, prefill 918 tok/s — the
35 tok/s target is NOT met, bounded by the fork's expert-gather read path
(~8.2 GB/s effective vs a 22.74 GB/s link) at a 0.53-0.66 cache hit rate.**
Rollback: commit `2339fae` (the qwen FP8 + MTP4 shape of 2026-09-29 → 09-30)
[[frank-archon-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
back; the R9V state dirs are on disk (`/home/dank/r9v-state`,
`~/.local/state/r9v`) but the payload GGUFs are not — `/home/dank/models`
holds only the iq4r `.rad` as of 2026-10-18; git tag
frank-archon-r9v-q8-20260929 is the last pre-cutover commit):
[[frank-archon-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
(same-day swap to the Q8_0 payload; state + payloads no longer on disk
2026-10-18, only the R9V state dirs remain): the R9V radiance-vLLM fork serving
Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS (92.39 GiB, 3 shards) +
blockfp8 MTP4 (depth 4) + F16 vision on BOTH R9700s — TP=2 tiered
experts (UVA pinned host copy, 62 hot + 160-slot LRU on rank 0, 428
static on rank 1), PLE n-gram on ssd-mmap, CED on, 196608 ctx. Cut over
from llama.cpp the same day; rollback target
[[frank-archon-llamacpp-qwen38-flash-next-q8]]
- [[frank-archon-llamacpp-qwen38-flash-next-q8]] — RETIRED 2026-09-27
(rollback target; the llama.cpp b11030 endpoint of 2026-09-26):
unsloth llama.cpp (b11030-mix prebuilt, ROCm gfx120X, bundled runtime)
serving unsloth/Qwen3.8-Flash-Next-GGUF at Q8_0 across both R9700s —
routed experts and the 51B-param n-gram table computed in host RAM
(`-ot exps=CPU,per_layer_token_embd.weight=CPU`), the shared Q8_0 MTP head
(depth 2) chained behind ngram-mod, BF16 vision, one full 262,144-token
conversation, `reasoning_content`, the placement law, the cutover and
rollback
- [[frank-archon-lucebox-ds4]] — RETIRED 2026-09-26 (same-day swap,
rollback target; active for hours): the Lucebox DS4.1 engine
(localhost/lucebox-rocm:gfx1201) serving
Lucebox/DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB) with streamed experts —
the fleet's only non-llama.cpp runtime detour
- [[dankserver-frontier-moe-feasibility]] — VERIFIED 2026-09-28: which of
DeepSeek-V4.1-Flash (552B), MiMo-V2.6-Flash-MOPD (309B) and GLM-5.3-Flash
(320B) can run on the 2x R9700 box, on what runtime, and why V4.1 does not
fit (475 GiB payload, 189 GiB of it Engram; decode is PCIe-expert-stream
bound, ~23 tok/s ceiling at today's residency). Includes the Affinity
V4-Flash-to-V4.1 port assessment — reusable expert tiering, four
architecture gaps, one of them a silent-wrongness trap on the Engram key
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — VERIFIED 2026-09-29:
the flag-level decode-tuning campaign on the ad hoc qwen-fp8 container
(tcclaviger RDNA4 fork, orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8,
TP=2 across both R9700s). **>35 tok/s MET: 37.44-40.57 tok/s with
`--expert-prefetch` DELETED** — telemetry measured the prefetcher at
36% precision, wasting ~half the gather traffic (the LRU itself runs at
82.45% hit). Prefetch-on incumbent 26.8 survived every other challenger
(geometry, spec depth, TunableOp-corrupts, batch knobs, concurrency,
ngram-fork-bug); the residual gap to ~60 tok/s is the fork's gather
read path (6.5-9.5 GB/s effective vs 22.7 at the same slab size)
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26 (replaced by
[[frank-archon-lucebox-ds4]], then
[[frank-archon-llamacpp-qwen38-flash-next-q8]]): vllm-radiance 0.9.3
serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark
W4A8) with the z-lab DFlash2-FP8 drafter as ONE tensor-parallel process
(TP=2) behind :7770, on unquantized bf16 KV (`KV_DTYPE=auto`, cut
2026-09-21). The FP8 checkpoint is the on-disk rollback. The three-arm
battery that chose TP=2, the auto-sized KV pool, and the verified
capability battery (tools, reasoning)
- [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium:
the streamlining toolchain (hf-manifest.py, hf-fetch.sh,
swap-frank-medium.sh), the quantization-aware entrypoint contract, the
quadlet edit, the deploy, the capability battery, and the rollback path.
The 2026-09-20 MXFP4 → FP8 swap is the worked example
- [[frank-medium-llamacpp-dflash2]] — RETIRED 2026-09-16 → 2026-09-20: mainline
llama.cpp serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the
DFlash2 Q4_K_M drafter as ONE layer-split process (32:33 over two
163,840-token slots); superseded by the radiance TP=2 cutover, and the
rollback target. Keeps the batch/ubatch optimum curve and the per-layer split
arithmetic
- [[Dual-R9700-Cooperation]] — whether the two R9700s can serve ONE model: the
RCCL/PCIe-atomics wall on the old TUF GAMING B650-E, the 2026-09-20 ProArt
B850-Creator x8/x8 swap that cleared it, and the measured three-way verdict
(radiance TP=2 95.8 t/s decode vs layer split 45.7 vs llama tensor mode 30.5)
that put TP=2 in production
- [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era,
2026-09-15): charlie12345/ROCmFPX serving
kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the
runtime it replaced
- [[freetoken-frank-small]] — RETIRED 2026-09-15, when the RTX 3090 moved to
frank-dragoon: the FreeToken v0.1.2 frank-zealot runbook (batsclamp
Huihui-Qwen3.6-35B-A3B-abliterated-FP8), kept as the retirement record. The
NAME was reused on 2026-09-16 by a different service on Dankclaw — see
[[frank-zealot-qwen38-gsq-rco]]
- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
- [[deployment]] — deploy and update process for both servers
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
- [[frank-dragoon-rocmfpx-q6agent-uncensored-mtp]] — **RETIRED 2026-10-03 →
10-05 (rollback target; payload, fork build and build script on disk)**:
the pinned ROCmFPX fork serving the uncensored protected-heads
Q6_0_ROCMFPX_AGENT rung with the repo's MTP-Q4_0 head drafting 4 tokens,
one 262,144 window, f16 KV; measured 27.6-28.1 t/s prose decode at 86.8 %
acceptance. Replaced by the halogen DFlash2 endpoint; the ngram-mod-onto-MTP
rejection (-15 %, recurrent-state rollback) stays a live lesson for this
architecture.
- [[frank-dragoon-rocmfpx-q8agent-mtp]] — **RETIRED 2026-10-02 → 10-03 (the
rollback target; payload still on disk)**: the same pinned ROCmFPX fork
serving the ALIGNED `Q8_0_ROCMFPX_AGENT` (ftype 115, 17.0 t/s, MTP
acceptance 0.462) — the MTP-vs-none +123 % A/B, the trap catalogue
- [[frank-dragoon-gufo-qwen38-27b]] — **RETIRED 2026-10-02 (history; the
payload dirs were removed from disk 2026-10-03, so this rollback is a
re-download)** (root systemd unit, port 7770): gufo
(gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from
source on Dankclaw, serving unsloth/Qwen3.8-27B-GGUF **UD-Q4_K_XL**
with the z-lab DFlash2 Q4_K_M drafter — ONE session at the full native
262,144 window, unquantized f16 KV. Rolled back by Frank after one day
of [[frank-dragoon-vllm-mimo-v26-bf16]]. The two from-source build
workarounds, the client-contract delta, the measured prefill/decode
trade against llama.cpp
- [[frank-dragoon-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (one day;
rollback by Frank after the autoresearch loop): the digest-pinned
kyuz0/vllm-therock-gfx1151 image serving
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16, 4 x 262144 ctx,
CROSS-MODEL MTP from Qwen/Qwen3.5-9B — kept for the measured record:
battery decode 6.19 t/s (MTP-2) vs 9.01 t/s (no draft — spec NET-
NEGATIVE on real chat+reasoning traffic), the roof math, the greedy-text
numerics trap, and the death of every ≥6-bit vLLM quant path on gfx1151
(quark int8 rejected, FP8 kernels CDNA-only, torchao unwired)
- [[frank-dragoon-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
[[frank-dragoon-gufo-qwen38-27b]]; rollback target): stock llama.cpp
(unsloth b11030, ROCm gfx1151) serving
unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2
Q4_K_M drafter (depth 4) on the gfx1151 iGPU — one full 262,144-token
conversation, q8_0 KV (the build lacks fp8), text-only, Qwen-Sharp
v22.4.0 template, the 114 GiB GTT memory model, client contract,
troubleshooting
- [[frank-dragoon-kairic-edge-qwen38-27b]] — RETIRED 2026-09-21 (same-day
swap, replaced by [[frank-dragoon-unsloth-qwen38-27b-q8xl]]; rollback
target): the Kairic Edge runtime (ciru-ai/ROCmFPX v1.2, built from
source) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (IU4 W4A4, 27B dense)
on the gfx1151 iGPU — one full 262,144-token conversation, MTP depth 4 +
n-gram drafting, the three PromptForge sidecars, compatibility mode, the
114 GiB GTT memory model, client contract, troubleshooting
- [[frank-dragoon-unsloth-qwen38-flash-next]] — RETIRED 2026-09-21 (same-day
swap, replaced by [[frank-dragoon-kairic-edge-qwen38-27b]]; rollback target):
the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving
unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (125B MoE) on the gfx1151 iGPU —
one 200,000-token conversation, MTP depth 2 via the explicit -md shared-Q8_0
head, the 114 GiB GTT memory model, client contract, troubleshooting
- [[frank-dragoon-ciru-orca]] — RETIRED 2026-09-21 (replaced by
[[frank-dragoon-unsloth-qwen38-flash-next]]; rollback target): the CIRU
v4.4.1 runtime (private host build, QSA hotfix) serving
jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 iGPU — one
262,144-token conversation, MTP depth 4, 4096 MiB PLE cache, runtime-root
gate, client contract, troubleshooting (2026-09-19 → 09-21)
- [[halogen-flash-server]] — RETIRED 2026-09-19 (replaced by
[[frank-dragoon-ciru-orca]]; rollback target): peonist-ai halogen-flash-server
0.5.8 serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b
.hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one
400,384-position pool, deployed flags, memory model, client contract,
troubleshooting (2026-09-16 → 09-19)
- [[frank-zealot-llama-mimo-q6k]] — **RETIRED 2026-10-01 → 10-02** (the same
rung returned ACTIVE 2026-10-03 at the 2 x 200,704 f16-KV shape —
[[frank-zealot-llama-mimo-q6k-2x200704]]): llama.cpp (master, CUDA sm_86
build) serving
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** on the RTX 3090 — TWO
native 262,144-token sessions with ASYMMETRIC KV (K f16 unquantized + V
q8_0, which forces flash-attn), ~20.9/24.6 GiB measured, text-only (the
GGUF has no vision), reasoning_content, no spec decode; payload still at
`/home/dank/models/MiMo-Q6K/`
- [[frank-zealot-sglang-mimo-v26]] — RETIRED 2026-10-01 (hours; replaced
by [[frank-zealot-llama-mimo-q6k]] for the asymmetric KV SGLang cannot
express; venv still on disk, INT8 payload staged to
`~dank/.AGENTTRASH/20261002-retired-llm-payloads/`): SGLang in a
self-contained python3.12 venv serving leteschlead MiMo-V2.6 INT8
W8A16 AutoRound, 2 x 262144 ctx, fp8 KV, in-checkpoint vision,
83.9 tok/s decode
- [[frank-zealot-gemma4-12b]] — RETIRED 2026-09-17 → 2026-10-01 (replaced by
[[frank-zealot-sglang-mimo-v26]] → the llama.cpp MiMo shape; payload staged
to `~dank/.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/`):
mainline llama.cpp serving
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
3090, with the separate MTP assistant sidecar chained behind ngram-mod,
q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB.
- [[frank-zealot-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-
token pool, 2 concurrent full-context sessions, native text+image+audio.
- [[frank-zealot-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): the
prior frank-zealot runbook (2026-09-16 → 09-17), ISTA-DASLab
Qwen3.8-27B-GSQ-RCO IQ3_S with the model's own in-file MTP head and
f16 KV at 131k
- [[x4-pair-experiment]] — MEASURED 2026-09-16: what mainline llama.cpp can and
cannot do with the 3090 + Strix Halo pair, over their only link (the 3090's
Gen4 x4). The tensor inventory that decides any split, the single flag (`-ot`)
that kills per-tensor placement across devices, the two-device layer splits
that work (+7 % decode at 64k, cross-vendor included), the harness-parsing trap
that invalidated the first round, and the iGPU-only result that matches
halogen's deployed decode rate — at a memory cost the co-tenants cannot pay
- [[frank-dragoon-ik-uncensored]] — RETIRED 2026-09-16 (rollback target): the
ik_llama.cpp runbook (root systemd unit, 2026-09-15 → 09-16) serving
mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090
(dense/attention/KV) + LPDDR (experts/PLE) hybrid; its measured envelope and
decode root-cause chain are the record of why the fleet went back to halogen
- [[dankclaw-iommu-translation-off]] — Dankclaw's IOMMU boot line: **`iommu=pt`**
since 2026-10-18, measured on this host at no prefill cost (+2.0 %, 1,616 t/s
cold), which restored the XDNA2 NPU; the `amd_iommu=off` era (2026-09-11) it
replaced ran on a 13-16 % figure measured upstream only; includes the AMD
PerfOpt research, what was deliberately not copied from the reference machine,
and the revert path via `scripts/set-dankclaw-bootline.sh`
- [[kingjones-strix-rocmfp4]] — PRIOR frank-dragoon runbook and rollback
target: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params
(56B MoE + 51.2B PLE), single 113.47 GiB GGUF, on the card's pinned
kingjones30/ROCmFPX fork (2026-08-30 → 2026-09-11)
- [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results
- [[frank-medium-batch-ubatch]] — batch/ubatch sweep on the 2x R9700 layer split
(2026-09-16): the ubatch optimum at 512, why the logical batch is free, and
the split rebalance that followed it
- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model
on Dankclaw (retired)
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on
Dankserver (frank-zealot, retired 2026-09-02)
- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for
frank-zealot (2026-08-26): q8_0 + Hadamard KV cache,
split-mode none (single-GPU), mandatory FA_ALL_QUANTS build flags
- [[kingjones-strix-mtp]] — the former Q6 AGENT runbook (restored 2026-09-01
after Q8_0 failure, then superseded by the STRIX deployment on 2026-08-27)
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime
on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment
(deployed and reverted 2026-08-22)
- [[Dankserver System Specs]] — the 2x R9700 box (frank-archon)
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-dragoon on
the flash-server v2 runtime since 2026-10-06, frank-zealot on the RTX 3090)
- [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint
on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as
the retirement record — the IOMMU/SVA err=-19 root cause and where
every dependent route moved
## CIRU Reference (Dankclaw)
- [[ciru-activefpx-promptforge]] — the RETIRED v3 model, ROCm 7.15 toolchain,
and build history (on Dankclaw; the gfx1151 slot now serves frank-dragoon's
flash-server v2-native payload — see [[frank-dragoon-flash-server-v2-native]])
- [[kairic-edge]] — Kairic Edge IU4 runtime: build runbook, binary
identity, and experiment summary (retired 2026-08-22)
## DFlash2 Reference (archived)
- [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment
(2026-08-21)
## vLLM Reference (historical)
- [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
- [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware
notes and sizing (this architecture family is what the retired frank-zealot
served in the Huihui abliterated FP8 variant — see
[[freetoken-frank-small]]; the Ornith-1.5-35B-A3B CRACK checkpoint
that preceded it is archived at [[dealign-ornith-crack]])
## Memory Management
- [[RAM CPU Offloading in VLLM]] — KV cache swap between GPU VRAM and system RAM
- [[Hybrid CPU GPU Processing Dankserver]] — NVIDIA GPU compute with system RAM offload
- [[Shared Memory Architecture Dankclaw]] — AMD unified memory and ROCm configuration
## Retired Topology (2026-08-17 redo)
- [[Qwen3.8 PP Deployment]] — the retired pipeline-parallel head + worker
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history (superseded by FastMTP)
- [[FastFlowLM NPU Inference]] — the retired FastFlowLM NPU endpoint
- [[frank-dragoon-benchmark]] — backend/spec benchmark behind the old head
- [[dankserver-vram-growth]] — VRAM/RAM growth behind the retired cuBLAS crash
- [[DFlash DSpark Speculative Decoding]] — retired DFlash/DSpark drafting
- [[Quantization by Hardware Architecture]] — which quant format fits which GPU/CPU
## Pipeline Parallelism (archive)
- [[Pipeline Parallelism/index]] — index of multi-node inference research. The PP
pair is retired. These pages are research history.
## Runtime Research (llama.cpp family)
- [[Llama Cpp Runtimes]] — mainline vs charlie12345/ROCmFPX vs ik_llama.cpp: positioning, quantization, server fit
- [[Llama Cpp Code Diff]] — code-level quantization diff of the three runtimes + server recommendations
- [[SGLang]] — the structured-generation serving engine; when it fits Dankserver (2x R9700) vs Dankclaw (Strix Halo)
## Cross-Cutting References (skill distillation, 2026-10-05)
One polished reference page per skill-cluster, distilled from the managed
skill library into the vault (see [log](../log.md) for the wave plan):
- [[Quantization Production]] — how we self-make quants per card
(imatrix/K_P ladders, radiance recipes, REAP/Heretic/abliteration
prior art), and what each runtime decodes natively
- [[GGUF Compatibility]] — what a GGUF's tensor histogram and header demand
of a loader; how the runtime is chosen by the file
- [[Llama Cpp Serving Sizing]] — slot/pool/KV sizing laws, boot-shape
laddering, llama-bench discipline
- [[Llama Cpp Multi-Device Splits]] — layer vs tensor splits, `-ts`/`-ot`
laws, drafter placement
- [[PCIe Link and Atomics]] — the link walls, atomics routing, and the
heterogeneous-pair verdicts
- [[RDNA4 Container Serving]] — bring-up recipes for RDNA4 containers
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan law, every arena
lever, and why free VRAM is not arena
- [[vLLM Boot Forensics and Config Interrogation]] — reading a boot that
died, and interrogating a running fork's config
- [[vLLM Porting and Compile Triage]] — porting checkpoints across vLLM
versions/forks; torch.compile/dynamo breakage triage
- [[LLM Bench Windows and Measurement]] — window discipline, honest A/B,
the window script laws
- [[LLM Performance Diagnosis]] — slow prefill/decode triage on live
endpoints, IBS attribution, prefix-cache routing
- [[Model Swap and Rollback]] — the swap procedure, payload preflight,
research verification, second-endpoint bring-up
- [[Tcclaviger Fork Ops]] — the fork's sweep harness, measured knob
verdicts, large-MoE recipe, compile-cache persistence
- [[Heterogeneous Split Triage]] — triage for unequal-accelerator split
proposals (the no-go verdict format)
- [[FreeToken Builds]] — the CUDA and ROCm build paths with their
dependency traps
These sit alongside the operational pages under
`../Operations/index.md` and `../Agent/index.md`.
## Related
- [Project knowledge base](../index.md) — the Podman app stacks on the same
two hosts. OpenWebUI (Dankclaw) consumes the on-host `frank-dragoon`
endpoint at `:7770`. The Gitea instance on Dankclaw hosts this repository.
- [Deployment Flow](../Architecture/Deployment%20Flow.md) — the service-scoped
git-based deploy with inference isolation
| Page | Owns |
|---|---|
| [[Dankclaw Improvements]] | This page collects concrete, sourced recommendations to improve Dankclaw |
| [[Dankclaw System Specs]] | Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference. |
| [[dankclaw-iommu-translation-off]] | Dankclaw boots iommu=pt since 2026-10-18 — identity DMA with the IOMMU armed, which restored the XDNA2 NPU at a… |
| [[Dankserver System Specs]] | Dankserver hardware and OS state since the 2026-09-20 board swap (ASUS ProArt B850-Creator WiFi NEO, 2x AMD Radeon AI… |
| [[dankserver-frontier-moe-feasibility]] | VERIFIED 2026-09-28 by measurement and source reading. Compares DeepSeek-V4.1-Flash, MiMo-V2.6-Flash-MOPD and… |
| [[dankserver-vram-growth]] | Investigation of the recurring cuBLAS crash on frank-zealot. What grows, what does not, and what to do next. |
| [[dealign-ornith-crack]] | dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with… |
| [[deployment]] | Scoped deployment and update process for the three inference services (frank-archon, frank-dragoon, frank-zealot). |
| [[DFlash DSpark Speculative Decoding]] | DFlash and DSpark are the two newest speculative decoding methods. |
| [[Dual-R9700-Cooperation]] | Why frank-medium's two Radeon AI PRO R9700s could not run tensor-parallel on the old B650-E board (PCIe atomic… |
| [[FastFlowLM NPU Inference]] | FastFlowLM (FLM), the AMD NPU-first LLM runtime for Ryzen AI XDNA2 chips. Fit analysis for the 50 TOPS NPU on… |
| [[FreeToken Builds]] | Building FreeToken (the official franksdank fork of FreeTok Tokenworks' freetoken vLLM-MTP runtime) on Dankclaw — the… |
| [[freetoken-benchmark-sweep]] | KV pool vs decode speed sweep across bs1/bs3/bs4/bs6 on Dankserver with Qwen3.6-35B-A3B-FP8 via FreeToken v0.1.2. |
| [[GGUF Compatibility]] | How to answer "will engine X load this GGUF?" from the file's header alone, why a custom tensor type forces the… |
| [[Heterogeneous Split Triage]] | Triage before building when someone asks to split a model across unequal accelerators on one host (24 GB GDDR6X +… |
| [[Hybrid CPU GPU Processing Dankserver]] | Dankserver uses NVIDIA GPU with system RAM for hybrid processing. |
| [[ling-3.0-flash-trial]] | `bailingmoe3-support` (commit `db4480bc8`, build b10411) at `/home/dank/llama.cpp-ling`. |
| [[ling-3.0-flash-vllm-plan]] | This page was the plan for a vLLM-ROCm deployment of `jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native` (INT4 W4A16, ~62… |
| [[Llama Cpp Multi-Device Splits]] | What llama.cpp can and cannot express across devices, the no-collective layer-split path, split balancing by… |
| [[Llama Cpp Serving Sizing]] | Slot-pool sizing against a headroom band, per-device VRAM accounting, prompt-cache/state sizing and the QSA… |
| [[LLM Bench Windows and Measurement]] | Stopping-and-restoring benchmark windows on the fleet LLM hosts (llama-bench sweeps, HTTP param benches, multi-arm… |
| [[LLM Performance Diagnosis]] | Probing why a served LLM is slow — the flat-depth probe, the expert-offload duty-cycle arithmetic, kernel-time… |
| [[Model Swap and Rollback]] | The full model-swap discipline for the LLM fleet — fit preflight, fetch and payload verification, runtime build/flag… |
| [[PCIe Link and Atomics]] | Link probing (idle downshift, path ceilings, CUDA/HIP bandwidth probes), the streaming-vs-compute ceiling arithmetic,… |
| [[Quantization by Hardware Architecture]] | This page maps LLM quantization formats to the GPU and CPU architectures that support them. |
| [[Quantization Production]] | — |
| [[qwen3.8-strix-optimizations]] | This page answers one question: **how do we make Qwen3.8-27B serve faster on |
| [[RAM CPU Offloading in VLLM]] | vLLM uses CPU RAM and swap space to extend GPU memory capacity. |
| [[RDNA4 Container Serving]] | Bringing a model larger than VRAM onto Dankserver's 2x R9700 (gfx1201) with the tcclaviger vLLM stack — quant-class… |
| [[RDNA4 MoE Expert Offload VRAM Budget]] | The expert-offload plan arithmetic on the tcclaviger fork — which terms a deploy can move, the util law, the… |
| [[Running Qwen3.6 27B FP8 on Dankclaw]] | Qwen3.6-27B is a dense vision-language model from the Qwen3.6 series. |
| [[Running Qwen3.6 35B A3B FP8 on Dankserver]] | Qwen3.6-35B-A3B is a Mixture-of-Experts vision-language model from the Qwen3.6 series. |
| [[Shared Memory Architecture Dankclaw]] | Dankclaw uses AMD Strix Halo APU with unified memory architecture. |
| [[swift15-campaign-worklog]] | — |
| [[Tcclaviger Fork Ops]] | Operating the tcclaviger RDNA4 vLLM fork on dankserver (shape G era, rollback target since the 2026-10-05 FreeToken… |
| [[ubatch-benchmark-large]] | Standardized ubatch sweep for the active kingjones Qwen3.8 ROCmFPX deployment. |
| [[vLLM Boot Forensics and Config Interrogation]] | Capturing a vLLM container boot failure's real root cause from journald (the container log is already gone),… |
| [[VLLM Index]] | This index contains all documentation for vLLM deployment on Dankserver and Dankclaw. |
| [[VLLM Overview]] | vLLM is a fast and easy library for large language model inference and serving. |
| [[vLLM Porting and Compile Triage]] | Forward-porting a model architecture to a new vLLM major (additive-only discipline, gap enumeration, shims that… |
| [[vllm_research_findings]] | vLLM v0.26.0 (latest stable, released July 25, 2026) supports DFlash speculative decoding with the `--spec-method… |
| [[x4-pair-experiment]] | What mainline llama.cpp can and cannot do with a 3090 + Strix Halo combination, measured 2026-09-16 on Dankclaw — the… |
serverAppConfigs-docs/LLM/kairic-edge.md
-327
View File
@@ -1,327 +0,0 @@
---
type: reference
title: Kairic Edge IU4 Runtime on Dankclaw
description: jcbtc/Qwen3.8-27B-IU4-Kairic-Edge served by the Kairic Edge CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Build runbook, binary identity, serving profile, and rollback.
tags:
- llama.cpp
- rocm
- dankclaw
- gfx1151
- kairic
- iu4
- promptforge
timestamp: 2026-08-22
---
# Kairic Edge IU4 Runtime on Dankclaw
> **Retired 2026-08-22.** This experiment was deployed, benchmarks were
> collected, and the service reverted to v3 the same day (operator
> decision). `frank-dragoon` serves v3 again. This page is kept as
> the retired experiment's build runbook and reference.
`frank-dragoon` (port 52625) briefly served
`jcbtc/Qwen3.8-27B-IU4-Kairic-Edge` through the Kairic Edge CIRU ROCmFPX
llama.cpp runtime. The runtime routes qualified prompt and multi-token
verification shapes through the RDNA 3.5 native
`V_WMMA_I32_16X16X16_IU4` matrix instruction. It was reverted to
[[ciru-activefpx-promptforge]] (v3) on 2026-08-22.
Performance summary of the experiment: the IU4 lane accelerated prefill
1.5-2.3x over v3 at all depths, but prose decode was 15-26% slower
(inherent to the release: the IU4 lane does not accelerate M1 target
decode). Reasoning-off fast-greedy mode was ~8% faster per token than
production chat, and roughly 2x faster per turn from halved output volume.
The v3-era experience, including the 400 errors from the argmax fast path
rejecting non-greedy requests, motivated the revert.
## Identity
The release pins every input. Verify these values before trusting a build.
| Input | Pinned value |
|-------|--------------|
| Runtime repo | ciru-ai/ROCmFPX tag `kairic-edge-qwen38-27b-v1`, commit `e97b324` |
| Composable Kernel | ROCm/composable_kernel `fdf4bb7fcc984811cef48ce817d89aac064b984a` |
| CK release patch | `patches/composable-kernel-gfx1151-iu4.patch`, SHA-256 `ea57bca3793d35d884fc156b6c0bc33d8e672e3bb49f66ff67338503029d6af5` |
| Certified toolchain | TheRock 7.15.0a20260718, AMD clang 23.0.0, GCC 13.3.0, CMake 4.4.2, Ninja 1.13.0 |
| Local toolchain | TheRock 7.15 SDK (ling3-ciru venv), amdclang++ 23.0.0git, GCC 15.2.0, CMake 4.2.3, Ninja 1.13.2 |
| Frozen certified binary | `llama-server` SHA-256 `0ece51fa4489e58b7c6942ec2aa59baadca897d8cf87f8e1dc4dc0b10a6efc49` |
| Local binary | `/home/dank/ciru-qwen38/ROCmFPX-kairic/build-kairic/bin/llama-server`, SHA-256 `d7625a0c14ac4158fd793c50ac0cefd8cdcbf76b5aca5cddafa09a353dc8f048` |
The guide states that a portable rebuild does not match the frozen
certified binary byte for byte. The local toolchain differs from the
certified one. The local binary reproduces byte-identical across local
rebuilds (see [Build reproducibility](#build-reproducibility)).
## Model artifacts
Download all four files into one directory. The three `.pfs` sidecars are
accelerator-specific prepacked weight views. The runtime requires all of
them.
| File | Size | SHA-256 |
|------|------|---------|
| `Qwen3.8-27B-IU4-Kairic-Edge.gguf` | 15.48 GiB | `360caf7381907c3eca7ac0afd1228efc016af747f3f38637fb1c7f94daabac2a` |
| `Qwen3.8-27B-Kairic-IU4-FFN.pfs` | 7.99 GiB | `adcbb90a7b429a30a2a39043366d68320d72e8b4816a0f498e882b2f80a2ba2b` |
| `Qwen3.8-27B-Kairic-IU4-GDN.pfs` | 1.88 GiB | `82f931316f1c895da104915dec4697163808d06f0e6b2dc027cee7aa3afc0f0e` |
| `Qwen3.8-27B-Kairic-IU4-GDN-Output.pfs` | 0.70 GiB | `3b07e7b176559e4402924ba0c368532fa6f02118a33c71e70974c809bf6208a3` |
Total payload: 26.05 GiB. Local directory:
`/home/dank/models/Kairic-Qwen3.8-27B-IU4-Edge/`. All four SHA-256 values
verified on 2026-08-22.
## Build runbook
```mermaid
graph TD
A[apt build deps] --> B[Clone ROCmFPX at pinned tag]
B --> C[Clone CK at pinned commit]
C --> D[Apply and verify CK patch]
D --> E[Generate ck/config.h: release gap]
E --> F[Run release build script]
F --> G[Build verification commands]
G --> H[Download and verify model]
H --> I[Smoke test]
```
### 1. Install build dependencies
```
sudo apt-get update
sudo apt-get install -y build-essential cmake git ninja-build pkg-config libssl-dev
```
### 2. Clone the immutable release
The local checkout is a git worktree of the pinned tag at
`/home/dank/ciru-qwen38/ROCmFPX-kairic`.
```
git clone https://github.com/ciru-ai/ROCmFPX.git
cd ROCmFPX
git checkout kairic-edge-qwen38-27b-v1
git status --short # must print nothing
```
### 3. Prepare Composable Kernel
```
git clone https://github.com/ROCm/composable_kernel.git third_party/composable_kernel
git -C third_party/composable_kernel checkout fdf4bb7fcc984811cef48ce817d89aac064b984a
git -C third_party/composable_kernel apply ../../patches/composable-kernel-gfx1151-iu4.patch
sha256sum patches/composable-kernel-gfx1151-iu4.patch # ea57bca3793d...af5
git -C third_party/composable_kernel diff --check
```
### 4. Generate `ck/config.h` (release gap)
The pinned CK commit ships only `include/ck/config.h.in` (CMake template).
The release script and the guide never generate `include/ck/config.h`. A
clean build fails at the first CK include without it. Generate it with CK's
own CMake configure, then copy it into the CK source include tree:
```
cmake -S third_party/composable_kernel -B third_party/composable_kernel/build-ck \
-G Ninja -DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER=/usr/bin/gcc -DCMAKE_CXX_COMPILER=/usr/bin/g++ \
-DCMAKE_HIP_COMPILER="$ROCM_PATH/bin/amdclang++" \
-DCMAKE_PREFIX_PATH="$ROCM_PATH"
cp third_party/composable_kernel/build-ck/include/ck/config.h \
third_party/composable_kernel/include/ck/config.h
```
Notes:
- The CK configure can fail after `config.h` generation. The generated file
is still valid. Check `build-ck/include/ck/config.h` exists.
- The `CK_USE_WMMA` flag inside `config.h` gates only CK instance-selection
code. The Kairic patched CK headers do not reference it. Local rebuilds
with the flag ON and OFF produce byte-identical binaries.
- The file is untracked in CK git. A `git clean` in the CK tree removes it.
Regenerate it before any rebuild.
- A full CK configure without a GPU target leaves `CK_USE_WMMA` undefined.
Pass `-DGPU_TARGETS=gfx1151` for the certified-style variant. Both
variants build an identical Kairic binary.
### 5. Build
The release script owns the certified CMake options.
```
export ROCM_PATH=<path to the 7.15 SDK _rocm_sdk_devel directory>
JOBS="$(nproc)" scripts/build-kairic-edge-gfx1151.sh
```
Local `ROCM_PATH` is the split SDK layout inside the retired ling3-ciru
venv:
`/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_devel`.
The serve script assembles `LD_LIBRARY_PATH` from the three split
directories (`_rocm_sdk_devel`, `_rocm_sdk_libraries`, `_rocm_sdk_core`) so
the 7.15 libraries precede the system ROCm 7.2.4.
### 6. Build verification
```
./build-kairic/bin/llama-server --version
./build-kairic/bin/llama-server --help | grep -A1 -- '--kairic-edge'
ldd ./build-kairic/bin/llama-server | grep -E 'not found' # must print nothing
```
Expected local output: `version: 79 (e97b324)`,
`built with GNU 15.2.0 for Linux x86_64`.
### 7. Download and verify model artifacts
Download the four files from
[jcbtc/Qwen3.8-27B-IU4-Kairic-Edge](https://huggingface.co/jcbtc/Qwen3.8-27B-IU4-Kairic-Edge)
and verify the SHA-256 values in the table above.
### 8. Smoke test
Run the guide's smoke test: health, `/v1/models`, two identical
`cache_prompt` completion requests, warm request must report restored
prompt tokens. Inspect the server log for route fallback, swap activity,
and HIP errors. Confirm the IU4 lane at boot:
```
{"record":"promptforge_init","mode":"iu4_ffn","wmma":"v_wmma_i32_16x16x16_iu4",...}
{"record":"promptforge_gdn_init","projection":"qkvz_iu4_hadamard",...}
```
## Build reproducibility
On 2026-08-22 a fresh rebuild followed the documented procedure from a
clean CK tree (pinned commit, patch re-applied, `config.h` regenerated via
CK configure). The rebuild produced a byte-identical binary:
| Build | SHA-256 |
|-------|---------|
| Original (first deployment) | `d7625a0c14ac4158fd793c50ac0cefd8cdcbf76b5aca5cddafa09a353dc8f048` |
| Fresh rebuild (clean CK tree) | `d7625a0c14ac4158fd793c50ac0cefd8cdcbf76b5aca5cddafa09a353dc8f048` |
The two builds used different `config.h` variants (differ only in
`/* #undef */` comment lines and the `CK_USE_WMMA` state). The identical
output proves that variant difference has no effect on this build. The
original manual `config.h` step is a neutral, minimal bridge over the
release gap.
The original build tree stays at
`/home/dank/ciru-qwen38/ROCmFPX-kairic/build-kairic-orig/` until the next
service restart confirms the fresh tree, then it is a deletion candidate
(move to `.AGENTTRASH`).
## Serving profile
The service runs the pinned runner profile plus the local chat
deviations below. Tunables live in
`scripts/frank-dragoon.env`; flags in `scripts/serve-frank-dragoon.sh`.
See [[deployment]] for the update process.
| Setting | Value | Origin |
|---------|-------|--------|
| Bind | `0.0.0.0:52625` | Local (runner default `127.0.0.1:8080`) |
| Context | 262144, 1 slot | Pinned runner |
| Batch / ubatch | 2048 / 512 | Pinned runner (fastest measured prefill) |
| Model loading | `--no-mmap` | Local (benchmarks: no perf difference) |
| Threads | 16 target / 32 batch | Pinned runner |
| KV | f16 target and draft | Pinned runner |
| Prompt cache | `--cache-ram 8192 --cache-prompt --cache-idle-slots -ctxcp 32` | Pinned runner |
| Kairic Edge | `--kairic-edge` | Pinned runner |
| Speculative | `draft-mtp` depth 4, `p_split 0.10`, backend sampling | Pinned runner |
| Server sampling | temp 0 / top-p 1 / top-k 0 / min-p 0 | Pinned runner |
| Reasoning | on, deepseek format, budget -1, preserve_thinking | Local |
| `LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH` | 0 | Local (runner: 1) |
Local deviations and their reasons:
- **Bind/port**: fleet convention; the runner binds localhost only.
- **Reasoning on + deepseek**: thinking goes to `message.reasoning_content`,
matching the retired v3 service. `preserve_thinking=true` keeps thinking
tokens in the history across turns.
- **`--no-mmap`**: benchmarks show no performance difference versus mmap.
Kept per operator preference (2026-08-22).
- **Target greedy argmax fast path off**: the pinned runner sets
`LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=1`. That fast path fails closed on
any non-greedy request (400 error). The model card's own chat
instructions send temp 0.7 / top-p 0.8 / top-k 20 / presence_penalty
1.5, which the fast path rejects. Set to 0 for production chat on
2026-08-22. This does not affect the Kairic IU4 lane or the MTP draft
fast path (`LLAMA_MTP_CPU_ARGMAX_FASTPATH=1` stays on).
- **`--kairic-edge` auto-injects ngram-mod** (n_match 24, n_min 64,
n_max 64). This is release design, not a misconfiguration.
## Performance on this machine
Benchmarks from 2026-08-22 on scratch port 7893, 512 forced tokens,
greedy, production config unless noted. Stop the service before
benchmarking (see AGENTS.md).
| Context | Kairic prefill | Kairic decode | v3 prefill | v3 decode (200 tok) |
|---------|----------------|---------------|------------|---------------------|
| ~8k | 506 (runner) / 348 (prod) | 21.0 (runner) / 19.8 (prod) | 217.7 | 29.42 |
| ~16k | 439 (runner) / 370 (prod) | 23.5 (runner) / 19.4 (prod) | 211.2 | 26.42 |
| ~64k | 232.5 (prod) | 14.6 (prod) | 155.7 | 19.72 |
Findings:
- Prefill runs 1.5-2.3x faster than v3 at every depth.
- Prose decode runs 15-26% slower than v3 at every depth.
- `--no-mmap` makes no difference (19.89 vs 19.80 t/s decode). The
service now runs `--no-mmap` per operator preference.
- `-c 262144` versus `-c 8192` makes no difference to decode.
- Live service at ~116k context: 13.5-14.9 t/s, consistent with the 64k
bench. Long conversations run in the deep-context regime.
- MTP draft acceptance on prose: rate 0.31-0.35, mean accepted length
2.6-3.0. Draft fast path active, zero failures.
- The card's 47.73 t/s aggregate comes from a 164-task coding suite with
deterministic sampling and repeated coding context. It does not
represent prose chat. The card's natural-prose slice measured 34.88
t/s on the certified system.
### Batch profile comparison (2026-08-22)
Three batch profiles ran on the identical chat profile (fast path off,
reasoning on, `--no-mmap`; the ub2048 row was the earlier mmap run, and
mmap makes no difference):
| Context | b2048/ub512 decode (cold/warm) | b4096/ub2048 decode | b4096/ub4096 decode | ub512 prefill cold | ub4096 prefill cold |
|---------|--------------------------------|---------------------|---------------------|--------------------|---------------------|
| ~8k | 19.6 / 18.7 | 19.8 / 19.4 | 22.7 / 19.8 | 524 | 266 |
| ~16k | 21.8 / 21.5 | 19.8 / 18.8 | 18.1 / 22.5 | 485 | 258 |
| ~64k | 13.7 / 15.7 | 14.6 / - | 14.1 / 13.8 | 323 | 203 |
Ubatch is a prefill knob. Ubatch 512 gives the fastest prefill at every
depth, 60-90% above ubatch 4096. Decode is identical across all three
profiles within run-to-run noise. The IU4 lane accelerates 96-512-row
prompt and verification shapes; ubatch 512 matches those shapes, large
ubatches do not. Production runs the runner's qualified 2048/512.
## Rollback to v3
The v3 payload and binary stay on the machine for rollback.
1. Stop the service: `sudo systemctl stop frank-dragoon`
2. Restore the v3 scripts:
`git revert 4c43839 7aee1df 40d50fe` (or checkout the script pair from
commit `132e147`)
3. Deploy: `FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw`
4. Verify health and the `Output-K8` boot records.
v3 locations: model `/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/`,
binary `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`.
Full v3 runbook: [[ciru-activefpx-promptforge]].
## Citations
- Model card: <https://huggingface.co/jcbtc/Qwen3.8-27B-IU4-Kairic-Edge>
- Build and verification guide: <https://github.com/ciru-ai/ROCmFPX/blob/kairic-edge-qwen38-27b-v1/docs/kairic-edge-gfx1151.md>
- Pinned release: <https://github.com/ciru-ai/ROCmFPX/tree/kairic-edge-qwen38-27b-v1>
- Pinned runner: `scripts/run-kairic-edge-gfx1151.sh` in the release repo
## See Also
- [[ciru-activefpx-promptforge]] — retired v3 runtime, rollback target
- [[deployment]] — deploy and update process
- [[Dankclaw System Specs]] — hardware and memory model
- [[qwen3.8-strix-optimizations]] — Strix Halo optimization history
serverAppConfigs-docs/LLM/kingjones-q8agent-rocmfpx.md
+1 -1
View File
@@ -238,4 +238,4 @@ reads `--host/--port/--temp` as CLI flags, never as env vars.
262144 env context) are in place. Proven serving context today is 131072 at
`gl=45`. See the `set_rows` build bug section above.
- Preceding model: Kairic Edge Qwen3.8-27B IU4 (deployed 2026-08-25 ~05:42,
retired the same day for the AGENT model). See [[kairic-edge]].
retired the same day for the AGENT model). See [[frank-dragoon]].
serverAppConfigs-docs/LLM/kingjones-strix-mtp.md
-101
View File
@@ -1,101 +0,0 @@
---
type: reference
title: kingjones Qwen3.8-27B ROCmFP4 STRIX MTP on Dankclaw
description: kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Dense model with a standalone MTP Q4_0 draft head. frank-dragoon.
tags:
- llama.cpp
- rocm
- dankclaw
- gfx1151
- kingjones
- rocmfp4
- mtp
timestamp: 2026-08-24
---
# kingjones Qwen3.8-27B ROCmFP4 STRIX MTP on Dankclaw
> **RETIRED 2026-08-29.** This payload was `frank-dragoon` from 2026-08-25
> until the Q8_0 AGENT experiment on 2026-08-29 (it was restored again
> 2026-09-01 and served until 2026-08-30, when [[agentionai-fast-imatrix]]
> took over). The Q8_0 model was retired because it crashed before
> serving. Kept as the historical record of the Q6 AGENT model, standalone
> MTP draft, Q8 projector, and PromptForge runtime.
> This page documents a retired model and its runbook.
`frank-dragoon` (port 7770) serves
`Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP` from `kingjones777` on Hugging
Face. It is a **dense** Qwen3.8-27B, abliterated (uncensored), quantized to
the ROCmFP4 Q6 AGENT format (ftype 114). The MTP speculative head ships as
a **separate** Q4_0 draft GGUF, so a bundled-head auto-detect does not apply
here; the head is loaded with `--model-draft`.
## Model Files
The card ships six text files plus the vision projector. The service uses
the Q6 AGENT file (requested for the fleet), the separate MTP draft head
and the Q8 vision projector (`mmproj-Qwen3.8-27B-Q8_0.gguf`).
| File | Size |
|------|------|
| `Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf` | 23.15 GiB |
| `mtp-Qwen3.8-27B-Q4_0.gguf` (draft head) | 1.56 GiB |
| `mmproj-Qwen3.8-27B-Q8_0.gguf` (vision) | 0.59 GiB |
Store the files under `/home/dank/models/kingjones-Qwen3.8-27B-STRIX-MTP/`.
## Speculative Decoding
Dense model, so the finding measured on the dense Qwen3.8 family holds:
**MTP alone at `--spec-draft-n-max 4` is the knee**. ngram under MTP hurts
(measured -4 to -5 t/s on the v3; see [[ciru-activefpx-promptforge]]). The
draft head is a separate file loaded via:
```
--spec-type draft-mtp
--model-draft mtp-Qwen3.8-27B-Q4_0.gguf
--spec-draft-n-max 4
--spec-draft-n-min 0
--spec-draft-p-min 0.0
```
The draft head loads on `ROCm1` with full offload (`--spec-draft-ngl 999`)
and f16 KV.
## Launch Excerpt
The key flags on the CIRU ROCmFPX runtime:
```
llama-server \
-m Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf \
--mmproj mmproj-Qwen3.8-27B-Q8_0.gguf \
--chat-template-file chat-template.jinja \
--chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true}' \
-ngl 999 -dev ROCm0 -c 196608 -np 1 \
-b 8192 -ub 8192 -ctxcp 32 \
--spec-type draft-mtp --model-draft mtp-Qwen3.8-27B-Q4_0.gguf \
--spec-draft-ngl 999 --spec-draft-n-max 4 \
-fa on --no-mmap --cache-type-k f16 --cache-type-v f16
```
## Known Issues
1. **GPU only**: Vulkan, CUDA, and CPU cannot load these files.
2. `--spec-draft-n-max` defaults to 16; set 4.
3. `reasoning_effort: "none"` throws in the Qwen API. Use
`enable_thinking: false` instead.
4. Small `max_tokens` with thinking on returns empty `content`. The Sharp
template drives thinking on by default (effort medium).
5. `preserve_thinking:true` (in `chat_template_kwargs`) keeps `<think>`
blocks in recorded assistant messages. `preserve_thinking:false` strips
them. See [[deployment]] for the chat_format/reasoning_format pairing
(`peg-native` + `deepseek`) the Pi agent expects from this endpoint.
## See Also
- [[deployment]] — deploy and update process
- [[ciru-activefpx-promptforge]] — the retired v3 page
- [[Dankclaw System Specs]] — the box
- [[frank-redo-2026-08-17]] — fleet topology history
serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md
-322
View File
@@ -1,322 +0,0 @@
---
type: reference
title: kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw (prior, rollback target)
description: PRIOR frank-dragoon runbook and rollback target (2026-08-30 → 2026-09-11). kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF served by the card's pinned kingjones30/ROCmFPX fork. 177B params (56B MoE + 51.2B PLE streamed via mmap), single 113.47 GiB GGUF, 401408 ctx x 2 slots.
tags:
- llama.cpp
- rocm
- dankclaw
- gfx1151
- kingjones
- rocmfp4
- strix-halo
- qwen4exp
timestamp: 2026-08-30
---
# kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw
> **PRIOR / ROLLBACK TARGET since 2026-09-11** — frank-dragoon moved to
> [[halogen-flash-server]] (peonist-ai halogen-flash-server 0.5.8 serving
> Qwen3.8-Flash-Next w4b), which upstream measures at roughly 4x this
> runtime's prefill. Everything below documents the llama.cpp era and stays
> accurate for a rollback: the payload is still at
> `/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX`, the binary
> at `/home/dank/GitPackages/kingjones-ROCmFPX/build`, and the unit, env and
> serve script are staged in `serverAppConfigs/.AGENTTRASH/` (`.llama.cpp`).
> Rolling back also requires removing the halogen Quadlet from
> `~/.config/containers/systemd/` so it stops claiming port 7770.
>
> Ran 2026-08-30 → 2026-09-11 on the model card's **pinned runtime**: the
> `kingjones30/ROCmFPX` fork. Text-only tier; the agentionai f16 vision
> tower was paired for mmproj and loads clean. Speculative decoding is OFF
> (the card ships no spec flags; the Flash-Next MTP heads do not load on
> this loader family — see MTP Draft Head).
`frank-dragoon` (port 7770) serves
`Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf` (single file, 113.47 GiB)
from `/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/`
via `scripts/serve-frank-dragoon.sh` + `scripts/frank-dragoon.env`
and the Qwen-Sharp chat template.
## Model Architecture
Base model: Qwen3.8-Flash-Next (177B parameters total).
| Component | Params | Details |
|-----------|--------|---------|
| MoE experts | 125.8B | 512 experts, 10 used per token |
| Attention | dense | 48 layers (12 attn + 36 SSM) |
| PLE table | 51.2B | `per_layer_token_embd`, Q8_0 |
| lm head | ~0.3% | Q6_K |
The PLE (per-layer embedding) n-gram table is the only thing that differs
between STRIX, STRIX_LEAN, and FAST tiers. Everything else is byte-identical.
## File Details
| Field | Value |
|-------|-------|
| File | `Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf` |
| Size | 113.47 GiB |
| Format | Single GGUF (not sharded) |
| file_type | 105 (`Q4_0_ROCMFP4_STRIX`) |
| General | 113.47 GiB, 5.51 bpw |
Tensor composition:
| Tensor group | Type |
|---|---|
| MoE expert weights (`ffn_*_exps`, 144) | TYPE_101 (ROCmFP4, 4.251 bpw) |
| Shared expert (`ffn_*_shexp`, 144) | TYPE_101 |
| Attention (`attn_*`, 120) | half TYPE_100, half TYPE_101 |
| PLE (`per_layer_token_embd.weight`) | Q8_0 — 50.66 GiB |
| `token_embd.weight` | Q6_K |
| `output.weight` (lm head) | Q6_K |
| Norms / biases | F32 |
## File Type Reference
| Tier | file_type | PLE table | Total size |
|------|-----------|-----------|------------|
| FAST | TYPE_106 | 25.34 GiB | 87.94 GiB |
| STRIX_LEAN | TYPE_105 (old STRIX_LEAN) | 35.76 GiB | 98.49 GiB |
| **STRIX** | **105 (`Q4_0_ROCMFP4_STRIX`)** | **50.66 GiB** | **113.47 GiB** |
TYPE_100 = 4.506 bpw, TYPE_101 = 4.251 bpw (measured exactly by solving
three files for two unknowns).
## Memory
The PLE table never enters GPU VRAM. It streams from the file through the OS
page cache.
| Context | GTT at load | GTT after prompt | GPU resident |
|---------|-------------|-------------------|-------------|
| 8,192 | 63.3 GiB | 63.6 GiB | ~63 GiB |
| 32,768 | 64.1 GiB | 64.8 GiB | |
| 65,536 | 65.1 GiB | 66.2 GiB | |
| 131,072 | 67.2 GiB | 69.1 GiB | ~69 GiB |
~63 GiB is the non-PLE mass plus compute buffers. **The PLE table does not
grow GPU memory regardless of context depth.**
Three critical consequences:
1. **`--no-mmap` gets you OOM-killed.** It forces the PLE into anonymous memory
(neither file-backed nor reclaimable). The cgroup killer takes the process
with nothing in the server log — only `dmesg` shows the evidence.
2. **Don't force PLE to CPU with `-ot`.** Measured generation dropped from
~23 to 13.4 tok/s. The kernel streams it better than any offload hack.
3. **A bigger tier costs disk, not GPU memory.** If you have storage, take
STRIX.
## Performance
Measured on Ryzen AI MAX+ 395 (gfx1151, ROCm 7.2.4), 49/49 offload. Prompts
sized to 85% of context window:
| Context | Prompt tokens | pp tok/s | gen tok/s |
|---------|--------------|----------|-----------|
| 8,192 | 6,963 | 279 | **22.02** |
| 32,768 | 27,852 | 264 | **20.52** |
| 65,536 | 55,705 | 234 | **18.23** |
| 131,072 | 111,411 | 185 | **15.34** |
Short prompts (~3,300 tokens): 381 tok/s prompt processing, 22.7 tok/s
generation (median of 3 runs).
## Benchmark: threads-batch tuning
Sweeped `--threads-batch` (16, 32, 48, 64, 80, 96, 128) with a 64K-token
prompt via HTTP API (`max_tokens=1`). Fresh server start per value.
| threads-batch | prefill tok/s | wall time |
|---------------|--------------|-----------|
| **16** | **373.5** | 62.9s |
| 32 | 354.4 | 66.3s |
| 48 | 344.1 | 68.3s |
| 64 | 344.7 | 68.2s |
| 80 | 344.4 | 68.2s |
| 96 | 357.1 | 65.8s |
| 128 | 321.1 | 73.2s |
**Optimal: `--threads-batch 16`** (373.5 tok/s). Lower thread batching wins
for prefill on this model -- fewer threads compete for GPU resources during
the prompt processing phase. The default of 32 is 5.1% slower. Above 64,
performance plateaus then declines (128 is 14% worse than optimal).
## Benchmark: decode throughput
Sweeped `--threads-batch` (8, 16, 32) with a ~23.5K-token code prompt and
256 generated tokens via HTTP API. Fresh server start per value.
| threads-batch | gen tokens | wall time | gen tok/s |
|---------------|-----------|-----------|-----------|
| 16 | 256 | 74.8s | 3.4 |
| 32 | 256 | 75.0s | 3.4 |
| 8 | 256 | 76.6s | 3.3 |
**Decode throughput is essentially flat across all tested values.** TB=16 and
TB=32 are identical at 3.4 tok/s; TB=8 is marginally slower at 3.3 tok/s.
Combined with the 5.1% prefill advantage at lower thread counts, `--threads-batch
16` is the optimal setting for both phases.
## Runtime build (kingjones30/ROCmFPX)
The card pins its own fork: [kingjones30/ROCmFPX](https://github.com/kingjones30/ROCmFPX)
— charlie12345/ROCmFPX plus the `qwen4exp-on-rocmfpx-d3ca537` patch, so
ROCmFP4 tensor types and the qwen4exp arch live in one tree. Built at
`/home/dank/GitPackages/kingjones-ROCmFPX/build` from commit `36e9acd`
(2026-08-27 card revision) with the card's recipe:
```bash
git clone https://github.com/kingjones30/ROCmFPX.git
cd ROCmFPX
cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release -DLLAMA_BUILD_WEBUI=OFF
cmake --build build --target llama-server llama-quantize -j16
```
- **`-DLLAMA_BUILD_WEBUI=OFF` is required on this box**: the tree
(post-WebUI llama.cpp) provisions embedded UI assets via npm or the
HF llama-ui bucket; npm is absent here and the bucket download
half-failed, aborting the link at 98%. The API server is unaffected.
All other compile work succeeded with zero errors, confirming the
card's clean-clone claim.
- All 35 flags our serve script passes were verified present in this
tree (`--spec-type`, `--cache-idle-slots`, `--ctx-checkpoints`,
`--reasoning-budget`, `--mmproj`, …).
## Serving Profile
Deployed args (`scripts/frank-dragoon.env` — our validated
profile, unchanged from the FAST deployment except the binary and model):
```
llama-server \
--model Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf \
--host 0.0.0.0 --port 7770 --jinja \
--device ROCm0 --gpu-layers 999 \
--fit off --flash-attn on \
--ctx-size 401408 --parallel 2 \
--batch-size 4096 --ubatch-size 512 \
--cache-type-k f16 --cache-type-v f16 \
--threads 16 --threads-batch 16 \
--ctx-checkpoints 16 --cache-ram 8192 \
--cache-prompt --cache-idle-slots --metrics \
--temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 \
--presence-penalty 1.5 \
--reasoning on --reasoning-format deepseek --reasoning-budget -1 \
--chat-template-file chat-template.jinja \
--chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}' \
--mmproj mmproj-Qwen3.8-Flash-Next-f16.gguf
```
Key choices:
- **`--fit off`** — auto-fit segfaults on this model's qwen4exp
hyper-connection/indexer configuration.
- **mmap (default)** — do NOT use `--no-mmap`. See memory section.
- **`--flash-attn on` works unpatched on this model** — verified live
2026-08-30 (graph built, model served). The QSA F16 `ggml_fill`
assert that forced [[agentionai-fast-imatrix]] onto the patched
charlie tree did not fire for the kingjones STRIX file; if a future
build aborts with `GGML_ASSERT(a-type == GGML_TYPE_F32)`, port
`scripts/patches/qwen4exp-qsa-f32-mask.patch` into this tree.
- **`--ctx-size 401408 --parallel 2`** (200704/slot) — carried from the
FAST deployment; the card's own ladder proves the window is nearly
free (QSA caps KV: ~8 GiB total 8k→256k). Verified live at depth:
a 111k-token prompt prefills at ~230 t/s while a second slot decodes.
- **`--ubatch-size 512`** — card re-measurement: -ub 1024/2048/4096
show no effect; 512 is already right on this hardware.
- **`--mmproj`** — the STRIX repo ships no vision tower; the agentionai
FAST package's f16 mmproj (same base model) is paired and loads
(one benign `warmup: CLIP graph uses unsupported operators` warning).
- **No spec flags** — speculative decoding is off; see below.
## MTP Draft Head — not loadable on this loader family
The previous draft of this runbook ran
[EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF](https://huggingface.co/EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF)
(Q8_0, 4.1 GB) as `--model-draft`. It does **not** load: the qwen4exp
loader requires a full per-layer `blk.<il>.hc_attn_norm.weight` set and
every Flash-Next-era head carries only `blk.48` (verified 3x against
multiple mains on 2026-08-28/29, see [[agentionai-fast-imatrix]] known
issues). The service therefore runs with no speculative decoding — the
card's own Usage block ships none. The card's published +50% decode
figure was measured with `Aristo94/EngramHalo.cpp`, a fork we do not
build. Revisit only if a head with the complete hyper-connection norm
set (or a loader that tolerates partial draft tensors) appears.
## Validation (2026-08-30 swap)
1. Integrity: on-host file `sha256sum` = `c6770d7442a06bf1d78edf28cec83e1ec93afdd34664c23ff898807b6b9349fa`,
byte-identical to the HF LFS oid. No re-download needed.
2. Load: systemd unit healthy; model loaded in ~225 s; health 200.
3. Generation: `/v1/chat/completions` returns correct
`reasoning_content` + content through the Sharp template (deepseek
format, `preserve_reasoning`).
4. Real load: an operator session's ~111k-token prompt prefilled at
213–232 t/s under the new runtime, matching the card's 131k rung
(196 t/s), with checkpoints + QSA KV streaming live.
## Deployment
### Download
```bash
# on Dankclaw
TOKEN=$(cat ~/.cache/huggingface/token)
DIR=/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX
mkdir -p "$DIR"
curl -fsSL -C - -o "$DIR/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf" \
--retry 5 --retry-delay 3 --retry-max-time 7200 \
-H "Authorization: Bearer $TOKEN" \
"https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF/resolve/main/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf"
```
The file is 113.47 GiB. Allow ~20-30 minutes for download depending on network.
### Swap procedure (as run 2026-08-30)
1. Build the runtime (see Runtime build) BEFORE touching the service.
2. Point the model file's directory at the canonical path (a plain `mv`
— never delete a 113 GiB payload to re-fetch it).
3. Edit `scripts/frank-dragoon.env` (+ serve script defaults) via
git: commit, push, `./scripts/deploy.sh dankclaw --service
frank-dragoon`.
4. Verify health: `curl http://192.168.0.143:7770/health` and a real
`/v1/chat/completions` probe.
Rollback: the FAST deployment is one env commit away (`LLAMA_BIN`,
`MODEL_DIR`, `MODEL_FILE`, `MMPROJ_PATH`); the charlie `build-qwen38`
binary and the agentionai payload remain on disk.
### Config files
- Env: `scripts/frank-dragoon.env`
- Serve: `scripts/serve-frank-dragoon.sh`
- Chat template: `scripts/chat-template.jinja`
## Known Issues
1. **Pinned fork required.** Stock llama.cpp and plain charlie12345/
ROCmFPX each lack half the tree (qwen4exp arch / ROCmFP4 types). Use
kingjones30/ROCmFPX, or the equivalent merged charlie tree
(`/home/dank/GitPackages/ROCmFPX`, PR #27742 commits).
2. **Single-file payload.** Not sharded. Use `hf download` or a resumable
curl. Browser download unreliable for 113 GiB.
3. **Model load is slow.** A few minutes cold for the 113 GiB file with a
fresh KV cache. Health returns 503 while loading. Do not kill the process.
`deploy.sh` waits up to 20 minutes.
4. **Kernel spam about `amdgpu: SVM mapping failed`** is a known documented
symptom, not root cause. Investigate other causes before assuming memory
pressure.
## Citations
- [kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF)
- [Qwen/Qwen3.8-Flash-Next base model](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)
- [kingjones30/ROCmFPX](https://github.com/kingjones30/ROCmFPX) — the card's pinned fork (charlie12345/ROCmFPX + [qwen4exp-on-rocmfpx-d3ca537.patch](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF/resolve/main/qwen4exp-on-rocmfpx-d3ca537.patch), mirrored in the HF repo)
- [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) — upstream fork defining the ROCmFP4 formats; our rollback tree with PR #27742 merged
- [llama.cpp PR #27739](https://github.com/ggml-org/llama.cpp/pull/27739) — MTP draft head support
serverAppConfigs-docs/LLM/ling-3.0-flash-trial.md
+2 -2
View File
@@ -8,7 +8,7 @@ timestamp: 2026-09-03
# Ling-3.0-flash Q6_K Trial on Dankclaw
> **RETIRED 2026-08-17.** The `frank-dragoon-ling` unit is disabled under the boot policy. At this page's retirement date `frank-dragoon` ran the Qwen3.8-27B PP head (since superseded by the STRIX deployment — see [[kingjones-strix-rocmfp4]]), not the Qwen3.6-27B config described in the (obsolete) switch-back section below.
> **RETIRED 2026-08-17.** The `frank-dragoon-ling` unit is disabled under the boot policy. At this page's retirement date `frank-dragoon` ran the Qwen3.8-27B PP head (since superseded by the STRIX deployment — see [[frank-dragoon]]), not the Qwen3.6-27B config described in the (obsolete) switch-back section below.
## Status
@@ -111,6 +111,6 @@ re-enable `frank-dragoon-ling` from the repo unit
## See Also
- [[frank-dragoon-benchmark]]
- [[frank-dragoon]]
- [[DFlash DSpark Speculative Decoding]]
- [[deployment]]
serverAppConfigs-docs/LLM/qwen3.8-spec-matrix.md
+1 -1
View File
@@ -162,6 +162,6 @@ q8_0/f16 KV.
## See Also
- [[frank-dragoon-benchmark]]
- [[frank-dragoon]]
- [[ling-3.0-flash-trial]]
- [[deployment]]
serverAppConfigs-docs/LLM/qwen3.8-strix-optimizations.md
+4 -3
View File
@@ -15,7 +15,8 @@ and the test matrix planned for 2026-08-16 morning.
Related pages:
- [Speculative-decoding matrix](qwen3.8-spec-matrix.md)
- [Backend and spec benchmark](frank-dragoon-benchmark.md)
- [[frank-dragoon]] — the endpoint this host became; its Eras and Laws carry the
backend and spec-decode measurements.
- [Dankclaw Improvements](Dankclaw%20Improvements.md) (OS-level items)
- [Pipeline Parallelism: Qwen3.8 PP Deployment](Pipeline%20Parallelism/Qwen3.8%20PP%20Deployment.md)
@@ -253,5 +254,5 @@ production @16k) with the same 8-bit model.
`-dio` for RPC).
- llama.cpp b10433 source: `src/llama.cpp` (device selection),
`common/speculative.cpp` (MTP/ngram-mod), `ggml/src/ggml-rpc` (protocol).
- Local benchmarks: [frank-dragoon-benchmark](frank-dragoon-benchmark.md),
[qwen3.8-spec-matrix](qwen3.8-spec-matrix.md).
- Local benchmarks: the measurement record lives on [[frank-dragoon]] (Eras and
Laws) and in [qwen3.8-spec-matrix](qwen3.8-spec-matrix.md).
serverAppConfigs-docs/LLM/ubatch-benchmark-large.md
+1 -1
View File
@@ -53,4 +53,4 @@ from the new harness here after a clean run.
- [LLM fleet index](index.md)
- [Deployment](deployment.md)
- [Active kingjones runbook](kingjones-strix-mtp.md)
- [[frank-dragoon]] — the kingjones MTP era this sweep ran against is an Eras row there (its page was folded in 2026-10-18)
serverAppConfigs-docs/LLM/vLLM Boot Forensics and Config Interrogation.md
+2 -2
View File
@@ -203,5 +203,5 @@ clipping at depth) starts with sections 1–3 here before touching anything.
## Related
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan arithmetic behind OOMs
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — deployed shape and history
- [[frank-dragoon-benchmark]] and [[frank-zealot-benchmark]] — probe/ladder hygiene on live hosts
- [[frank-archon]] — deployed shape and history
- [[frank-dragoon]] and [[frank-zealot]] — probe/ladder hygiene on live hosts
serverAppConfigs-docs/LLM/vLLM Porting and Compile Triage.md
+2 -2
View File
@@ -326,5 +326,5 @@ consumed at `v1/worker/gpu/cudagraph_utils.py:621`).
- [[vLLM Boot Forensics and Config Interrogation]] — capture and contracts
- [[RDNA4 Container Serving]] — the serving stack being ported onto
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the live deployment
- [[halogen-flash-server]] — the retired w4b-era endpoint and its profiling notes
- [[frank-archon]] — the live deployment
- [[frank-dragoon]] — the retired w4b-era endpoint and its profiling notes
serverAppConfigs-docs/LLM/x4-pair-experiment.md
+2 -2
View File
@@ -248,8 +248,8 @@ distinguishes a real split from a mislabelled sweep.
## Conclusion
The fleet's answer is unchanged — **two independent endpoints**
([[halogen-flash-server]] on the iGPU via halogen,
[[frank-zealot-qwen38-gsq-rco]] holding a whole 27B in the 3090's 24 GB) — but two
([[frank-dragoon]] on the iGPU via halogen,
[[frank-zealot]] holding a whole 27B in the 3090's 24 GB) — but two
claims from the first round of this experiment are now corrected:
1. **Cross-vendor layer split is possible** (measured: CUDA0 + Vulkan1, 27.23
serverAppConfigs-docs/Research/Llama Cpp Code Diff.md
+1 -1
View File
@@ -211,7 +211,7 @@ a packing transform on the K-family — and is likewise a modifier, not a quant.
## See Also
- [[Llama Cpp Runtimes]] — positioning + quantization overview
- [[dealign-ornith-crack]] — the CRACK MoE on Dankserver
- [[kingjones-strix-mtp]] — the ROCmFP4 model on Dankclaw
- [[frank-dragoon]] — the endpoint the Dankclaw ROCmFP4 eras became (their payloads and shapes are Eras rows there)
- [[Quantization by Hardware Architecture]] — quant vs hardware
## Citations
serverAppConfigs-docs/Research/Llama Cpp Runtimes.md
+1 -1
View File
@@ -125,7 +125,7 @@ Serves **kingjones Qwen3.8-27B ROCmFP4 STRIX MTP** (the production build).
## See Also
- [[Llama Cpp Code Diff]] — the code-level quantization diff + comparison table
- [[dealign-ornith-crack]] — the CRACK model on Dankserver
- [[kingjones-strix-mtp]] — the ROCmFP4 model on Dankclaw
- [[frank-dragoon]] — the endpoint the Dankclaw ROCmFP4 eras became (their payloads and shapes are Eras rows there)
- [[Quantization by Hardware Architecture]] — which quant fits which GPU/CPU
- [[SGLang]] — the structured-generation serving engine (the other lane)
serverAppConfigs-docs/Research/SGLang.md
+1 -1
View File
@@ -142,7 +142,7 @@ SGLang-on-ROCm is supported (Docker/pip), but:
## See Also
- [[Llama Cpp Runtimes]] / [[Llama Cpp Code Diff]] — the llama.cpp side of the trio
- [[dealign-ornith-crack]] — the CRACK MoE on Dankserver
- [[kingjones-strix-mtp]] — the ROCmFP4 MTP model on Dankclaw
- [[frank-dragoon]] — the endpoint the Dankclaw ROCmFP4 MTP era became (an Eras row there)
- [[Quantization by Hardware Architecture]]
## Citations
serverAppConfigs-docs/index.md
+205 -536
View File
@@ -1,566 +1,235 @@
# Server App Configs
# serverAppConfigs — knowledge base
Podman deployment configuration for two homelab servers: Dankserver and Dankclaw.
One fact, one owner. A page owns exactly one thing and everywhere else links
to it with a wikilink. Shape, payload, eras, laws and rollback for an inference
endpoint live on its endpoint page; `scripts/<service>.env` is the authority for
every number, and a page that disagrees with it is the bug. Payload swaps are
rows in an endpoint page's **Eras** table, never pages of their own. This index
is a locator: one row, one clause, no numbers.
This repository is the single source of truth for both hosts. It tracks the
container stacks and the LLM inference fleet. Edits flow through this repo,
then deploy to the targets with `scripts/deploy.sh` (one script per target
covers both workloads).
## Inference endpoints
> 2026-08-22: the former `vllmScripts` repository (LLM fleet) is merged
> into this repo with its full git history. Its knowledge base now lives at
> [serverAppConfigs-docs/LLM/index.md](LLM/index.md).
| Page | Owns |
|---|---|
| [[Frank TTS]] | frank-tts is the fleet's cloned-voice speech endpoint on Dankclaw :7772 — the one voice surface consumers address, in… |
| [[frank-archon]] | The one frank-archon page: the live radiance shape and harness contract (read off scripts/frank-archon.env), the Eras… |
| [[frank-dragoon]] | The one runbook for frank-dragoon, the Strix Halo iGPU endpoint (192.168.0.143:7770): current shape from the env,… |
| [[frank-zealot]] | The one frank-zealot reference: the live shape on Dankclaw's RTX 3090 (:7771) exactly as scripts/frank-zealot.env… |
## Hosts
## Retired endpoints
| Host | IP | Role | OS |
|------|----|------|----|
| Dankserver | 192.168.0.235 | Media & library stack | Ubuntu 24.04 |
| Dankclaw | 192.168.0.143 | AI, search & development stack | Ubuntu 26.04 |
These services no longer exist, so no live page owns them; rollback provenance
is inside each page. Every payload swap of a LIVE endpoint is an Eras row on
that endpoint's page, not a page.
## Stacks
| Page | Owns |
|---|---|
| [[frank-medium-batch-ubatch]] | llama-bench sweep of --batch-size against --ubatch-size on frank-medium (2x Radeon AI PRO R9700, layer split) at the… |
| [[frank-medium-llamacpp-dflash2]] | The former ACTIVE frank-medium deployment since 2026-09-16 (model swapped 2026-09-19) — mainline llama.cpp on… |
| [[frank-medium-model-swap]] | How to swap the checkpoint behind the frank-medium endpoint (Dankserver :7770, vllm-radiance TP=2) — the streamlining… |
| [[frank-medium-radiance-mxfp4]] | ACTIVE frank-medium runbook — vllm-radiance serving hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 (ParoQuant MXFP6 W4A8,… |
| [[frank-medium-rocmfpx]] | charlie12345/ROCmFPX llama.cpp serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF as frank-medium on Dankserver… |
| [[frank-npu]] | The retired FastFlowLM endpoint on Dankclaw's XDNA2 NPU — what it served, why it died on 2026-09-11 with the… |
| [[frank-redo-2026-08-17]] | This page records the full redo of the inference fleet on 2026-08-17. |
| [[ikllama-ornith-dankserver]] | ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-zealot on the RTX 3090. Q8_0 model, KV K default f16 / V… |
| [[kingjones-q8agent-rocmfpx]] | **Status (historical, 2026-09-13):** was to deploy on Dankclaw. |
| [[qwen3.8-spec-matrix]] | This page records the speculative-decoding benchmark history and the |
- [[Dankserver Stack]] — RomM, Copyparty, Calibre, Calibre-Web, Caddy
- [[Dankclaw Stack]] — SearXNG (via Gluetun), ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Caddy
## Dankclaw services (192.168.0.143)
## Architecture
| Page | Owns |
|---|---|
| [[ComfyUI]] | ComfyUI is a node-based AI image generation engine. |
| [[ComfyUI Qwen-Image Workflows]] | Qwen-Image 2.1 workflows on Dankclaw: two proven txt2img workflows (art + legible text), the API recipe, and which… |
| [[Dankclaw Stack]] | Service inventory and routing overview for the Dankclaw AI, development, and observability host. |
| [[Frank TTS Engine]] | frank-tts-engine is the arm A TTS engine on Dankclaw: qwentts.cpp's resident tts-server with the 1.7 B base talker,… |
| [[Gitea]] | Gitea deployment on Dankclaw with its web UI reverse-proxied by the canonical Caddy service. |
| [[Gluetun]] | Gluetun is a lightweight VPN client container. |
| [[Hermes]] | Hermes is the self-improving AI agent from Nous Research. |
| [[Hermes-Dashboard]] | The Hermes web dashboard is a browser UI for managing the agent. |
| [[Hermes-WebUI]] | The hermes-webui container on Dankclaw — the API server that the Hermex iPhone app talks to, reachable over an SSH… |
| [[Marreta]] | The content-cleaning proxy (manualdousuario/marreta, Laravel/serversideup fpm-nginx) deployed as a rootless Quadlet… |
| [[Open Notebook]] | open-notebook is the fleet's podcast generator on Dankclaw: notes and sources in, a multi-speaker episode out —… |
| [[OpenWebUI]] | OpenWebUI is a self-hosted chat interface for LLMs. |
| [[Postgres]] | PostgreSQL is the database backend for Gitea. |
| [[SearXNG]] | SearXNG is a privacy-respecting metasearch engine. |
| [[SearXNG Providers]] | How Google and other upstream search engines get blocked behind the Gluetun |
| [[Seerr]] | Seerr is the unified request manager for the *arr stack (the maintained `seerr-team/seerr` successor to the archived… |
| [[Swap Configuration]] | The host swapfile on Dankclaw — 64 GiB at /swap.img on ext4, fstab-driven via the generated swap.img.swap unit.… |
| [[Valkey]] | Valkey is the cache and limiter store for SearXNG. |
| [[Voice Packs]] | How a frank-tts voice pack is built on Dankclaw: the three files every pack must carry, the 10-second source-material… |
- [[Network Architecture]] — Caddy reverse proxy, hostname routing, Gluetun VPN path
- [[Storage Policy]] — bind mounts, data locations, ownership
- [[Deployment Flow]] — service-scoped git deploy, target map, and health checks
- [[Testing Policy]] — the test seams and how to run them
- [[E2E Suite]] — the Playwright suite that drives the live guard chain
- [[Container Network Namespaces]] — which local services are addressable from each Podman network namespace on Dankclaw
## Dankserver services (192.168.0.235)
## Dankserver Storage
| Page | Owns |
|---|---|
| [[Calibre]] | Calibre is the e-book library manager that backs Calibre-Web. |
| [[Calibre-Web]] | Calibre-Web is a web application that browses, reads, and downloads eBooks from an existing Calibre library database. |
| [[Copyparty]] | Copyparty is a lightweight file server and media streamer that runs as a single Python binary. |
| [[Dankserver Stack]] | Dankserver (192.168.0.235) runs the media and library stack. |
| [[Entropy Volume]] | The Dankserver entropy volume is a RAID-backed LUKS filesystem mounted at |
| [[Jellyfin]] | Jellyfin media server on Dankserver — quadlet unit, entropy-volume data layout, ports, client access, and… |
| [[mariadb]] | MariaDB 11 database backend for RomM on Dankserver — quadlet unit, network alias, env-file secret handling, and… |
| [[Plex]] | Plex Media Server is a **containerized media server on Dankserver**: a podman |
| [[redis]] | Redis 7-alpine cache and task-queue backend for RomM on Dankserver — quadlet unit, network alias topology, and backup… |
| [[RomM]] | RomM (ROM Manager) is a self-hosted ROM library manager and player. |
| [[RomM-Scraping]] | This guide covers metadata providers, cover art, ROM naming conventions, emulator integration, and the scraping… |
| [[Tautulli]] | Tautulli is the statistics, monitoring, and notification companion for |
- [[Entropy Volume]] — command-line unlock and mount helper
## FrankBook-Pro (192.168.0.112)
## Dankserver Services
| Page | Owns |
|---|---|
| [[ARR-Stack]] | The *arr automation stack (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) — FrankBook-Pro only, every… |
| [[lidarr]] | Lidarr music manager in the *arr stack on FrankBook-Pro — live quadlet deployment, ports, volumes, and wiring facts. |
| [[prowlarr]] | Prowlarr indexer-management layer for the *arr stack on FrankBook-Pro — feed routes, sync model, API surface, and… |
| [[qbittorrent]] | qBittorrent download client for the *arr stack — pinned 4.6.7 on FrankBook-Pro, shares the Gluetun WireGuard netns,… |
| [[radarr]] | Radarr movie manager in the *arr stack on FrankBook-Pro — live quadlet deployment, ports, volumes, and wiring facts. |
| [[readarr]] | Readarr ebook/audiobook manager in the *arr stack on FrankBook-Pro — retired upstream, pinned image, parked qB… |
| [[sonarr]] | Sonarr TV manager in the *arr stack on FrankBook-Pro — live quadlet deployment, ports, volumes, and wiring facts. |
- [[RomM]] — retro game manager
- [[RomM-Scraping]] — metadata providers, cover art, naming conventions
- [[Copyparty]] — file sharing and streaming
- [[Jellyfin]] — media server (movies, TV, music)
- [[Calibre]] — e-book library engine
- [[Calibre-Web]] — e-book web frontend
- [[Plex]] — media server on Dankserver (lscr.io/linuxserver/plex container via
quadlet; CPU decode only — no GPU passthrough)
- [[Tautulli]] — Plex statistics and notifications (v2.18.1-ls245, quadlet unit)
- [[mariadb]] — RomM database backend
- [[redis]] — RomM cache/queue backend
## Workstations
## FrankBook-Pro Services
| Page | Owns |
|---|---|
| [[Noctalia and Niri]] | The noctalia v5 desktop shell on niri (workstation + blankpad) — config layers, remote IPC over ssh, custom-palette… |
The *arr stack and qBittorrent run on **FrankBook-Pro** (192.168.0.112, the
macOS mini) as quadlets inside its podman machine — not on Dankserver.
## Cross-host infrastructure
- [[ARR-Stack]] — automated media download overview
- [[sonarr]] — TV show management
- [[radarr]] — movie management
- [[lidarr]] — music management
- [[readarr]] — ebook/audiobook management
- [[prowlarr]] — indexer management
- [[qbittorrent]] — torrent download client
| Page | Owns |
|---|---|
| [[Alertmanager]] | Alertmanager receives firing rules from Prometheus, groups and dedupes them, and forwards each group as a webhook to… |
| [[Alloy]] | Alloy is Grafana's log and telemetry shipper. |
| [[Anubis]] | Anubis is the bot-challenge proxy that fronts the guarded public vhosts on |
| [[Authentik]] | Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the |
| [[authentik-postgres]] | The PostgreSQL 16 database for Authentik — the homelab's SSO / |
| [[authentik-server]] | The Authentik server — the core, the admin API, and the embedded Caddy outpost. |
| [[authentik-worker]] | The Authentik worker — background tasks: event processing, provider |
| [[Caddy]] | Caddy is the single reverse proxy for both hosts. |
| [[FrankBook-Pro]] | FrankBook-Pro — the Mac mini running the *arr stack + qBittorrent inside a podman machine VM, gluetun-gated |
| [[Grafana]] | Grafana visualizes the metrics that Prometheus collects from every service. |
| [[Homepage]] | A password-protected web site on Dankclaw lists the services that are up, with a short description of each and a link… |
| [[Homepage Design Grilling]] | The decision log for the homepage design. |
| [[Homepage Status Probes]] | The homepage TUI status page — how probe targets are chosen (host-netns vantage), the three false-down shapes, triage… |
| [[LAN-DNS]] | The homelab gives every service a name under `franksdank.cloud`. |
| [[Loki]] | Loki is the log aggregation backend of the observability stack. |
| [[Node-Exporter]] | Node-Exporter exposes host metrics (CPU, RAM, disk, network) to Prometheus. |
| [[Observability]] | The Dankclaw observability stack — Prometheus, Grafana, Loki, Alloy, node-exporter and Alertmanager (Signal… |
| [[Observability Provisioning]] | How Prometheus config/rules and Grafana datasources/dashboards are provisioned as code in serverAppConfigs — the… |
| [[Pi-hole]] | Pi-hole is the LAN DNS server for the homelab. |
| [[Prometheus]] | Prometheus collects metrics from every service on Dankclaw by scraping their `/metrics` endpoints, and feeds the… |
## Dankclaw Services
## Repo architecture and policy
- [[SearXNG]] — privacy metasearch (VPN-routed)
- [[SearXNG Providers]] — unblocking Google & other upstream engines behind the VPN
- [[Gluetun]] — the VPN tunnel container
- [[Valkey]] — SearXNG cache and limiter store
- [[ComfyUI]] — node-based AI image generation (ROCm GPU, gfx1151 build)
- [[ComfyUI Qwen-Image Workflows]] — the two proven Qwen-Image 2.1 workflows + what to add next (LoRAs, Edit, ControlNet, Wan video)
- [[OpenWebUI]] — chat UI for local LLMs
- [[Hermes]] — the Nous Research agent
- [[Hermes-Dashboard]] — the Hermes web management UI (config, keys, sessions)
- [[Hermes-WebUI]] — the Hermex server (loopback API for the iPhone app, SSH-tunneled)
- [[Gitea]] — Git hosting (migrated)
- [[Postgres]] — Gitea database backend
- [[Seerr]] — unified *arr request manager (guard-chained; fronts FrankBook-Pro's stack)
- [[Marreta]] — content-cleaning proxy (guard-chained; shares SearXNG's Gluetun egress)
- [[Open Notebook]] — podcast generator (scripts by frank-zealot, voices by frank-tts); repo-owned config, not deployed
- [[Frank TTS Engine]] — the arm A TTS engine (`qwentts.cpp` resident `tts-server`, 1.7 B talker) behind frank-tts on a private bridge; deployed on Dankclaw, CPU-only, publishes no port by design
- [[Frank TTS]] — the 7772 cloned-voice endpoint consumers address: voice naming, pack layout, why the upstream is a container name; deployed, and it is the harness `speech` role
- [[Voice Packs]] — how a frank-tts voice is built: reference.wav + the pinned engine's speaker artifacts, cut past the 10-second source-material gate
| Page | Owns |
|---|---|
| [[Container Network Namespaces]] | Which local services are addressable from which Podman network namespace on |
| [[Deploy State Triage]] | Reading and reconciling the deploy-state markers — stale marker vs live container, pending-service lockout,… |
| [[Deployment Flow]] | Service-scoped Git deployment for the two homelab hosts. |
| [[E2E Suite]] | `tests/e2e/` drives the **live fleet** through a real browser: fresh contexts |
| [[Network Architecture]] | One Caddy fronts both hosts. |
| [[Quadlet Lifecycle]] | The full Quadlet discipline on this fleet — ownership auditing, compose-to-Quadlet cutover, drift/crash-loop… |
| [[Stale Signal Audit]] | The audit battery for surfaces that rot without failing — unreachable probes, vacuous assertions, invalid checkers,… |
| [[Storage Policy]] | All persistent data uses **bind mounts**: a host directory mapped directly into the container. |
| [[Testing Policy]] | This repository tests its configuration at defined seams. |
## Infrastructure
## Operations and incidents
- [[LAN-DNS]] — name resolution, Pi-hole, router DNS hand-off
- [[Pi-hole]] — the LAN DNS server (quadlet, records, web UI)
- [[Caddy]] — reverse proxy and TLS
- [[Anubis]] — bot challenge proxy fronting the guarded vhosts (chain, policy, gotchas)
- [[Authentik]] — the homelab's SSO / OIDC provider (forward_auth)
- [[authentik-postgres]] — Authentik database service
- [[authentik-server]] — Authentik server + Caddy outpost service
- [[authentik-worker]] — Authentik background worker service
- [[Homepage]] — password-protected service index
- [[Homepage Design Grilling]] — homepage decision log and open questions
- [[Observability]] — Grafana, Prometheus, Loki, Alloy, node-exporter, Alertmanager: the stack, its quadlet deployment, its exposure, and what is still missing
- [[Grafana]] — dashboards (datasource and the LLM fleet dashboard provisioned as code since 2026-09-16)
- [[Prometheus]] — metrics collection (5 alert rules + 1 recording rule, delivered via Alertmanager since 2026-09-17)
- [[Alertmanager]] — alert grouping and Signal delivery (webhook → host-side relay → signal-cli)
- [[Node-Exporter]] — host metrics
- [[Loki]] — log storage
- [[Alloy]] — log shipping (replaces Promtail)
- [[FrankBook-Pro]] — Mac mini *arr + qBittorrent inside podman machine (gluetun-gated)
| Page | Owns |
|---|---|
| [[Container Host Reachability]] | The two podman/netavark reachability quirks on the fleet — containers cannot dial unpublished host ports (LAN IP or… |
| [[Fleet Audit and Posture]] | Three standing audits — monitoring completeness (what should exist vs what is merely down), vhost proxy health… |
| [[Fleet Ops Playbook]] | Incident-shaped fleet procedures — LLM endpoint down with an abandoned GPU holder, scratch vLLM boot windows… |
| [[Payload and Disk Reclaim]] | Standing procedure for disk pressure on a fleet host — measure all buckets first, classify against the runbook index,… |
| [[Remote Job Runner]] | Detached long-job execution on fleet hosts with sentinel files (job.sh) — why ssh tail-watchers fail structurally,… |
| [[SSH and Remote Access]] | Passwordless SSH bootstrap without sshpass/expect, remote execution against fish login shells, and diagnosing a… |
## Research
## Agent / harness
- [[Service Research Brief]] — consolidated subagent research behind these pages
- [[ComfyUI ROCm and Vulkan]] — live-verified GPU path for ComfyUI on Strix Halo
- [[Lemonade and sd.cpp]] — the ggml-family image-generation alternative
- [[Llama Cpp Runtimes]] — mainline vs charlie12345/ROCmFPX vs ik_llama.cpp: quantization + server fit
- [[Llama Cpp Code Diff]] — code-level quantization diff of the three llama.cpp runtimes
- [[SGLang]] — the structured-generation serving engine (llama.cpp/vLLM comparison + our fit)
- [[Backblaze Integration]] — off-site backup plan (future)
- [[Jellyfin]] — media server research brief
- [[ARR-Stack]] — automated download stack research brief
- [[Local TTS and Voice Cloning]] — creating a local voice from samples, the 2026 cloning landscape, the full audio stack, and a recommended homelab deployment
- [[AI Job Search and Resume Tools]] — self-hosted AI job-search + resume-formatting apps, local-LLM fit, recommended stack
- [[Most Popular Self-Hosted AI Apps]] — the 15 most popular self-hosted AI apps of Jun–Sep 2026, homelab fit, adoption best practices
| Page | Owns |
|---|---|
| [[Agent Docs and Edits]] | Two self-inflicted damage classes — structurally broken markdown after parallel agent edits, and botched hashline… |
| [[Command Center Dotfiles]] | The ~/.omp repo is itself the deployment — stow-equivalent packages, full-replica rsync transport, the pre-overwrite… |
| [[OMP Harness]] | The omp agent harness as configured on this fleet — compaction chain arithmetic, per-model sampler routing via… |
## Skill Distillation (2026-10-05)
## Research and methodology
The managed skill library distilled into the vault, one reference page per
skill-cluster. See [log.md](log.md) for the wave plan and results.
| Page | Owns |
|---|---|
| [[AI Job Search and Resume Tools]] | Self-hosted AI job-search and resume-formatting applications: the 2026 landscape, local-LLM fit, and a recommended… |
| [[Backblaze Integration]] | Off-site backup plan for the bind-mounted data on both hosts. |
| [[ComfyUI ROCm and Vulkan]] | Research into whether ComfyUI can use GPU acceleration — not just CPU — on the AMD Strix Halo APU (gfx1151) in… |
| [[Lemonade and sd.cpp]] | Why Lemonade Server can do image generation on Strix Halo when ComfyUI cannot. |
| [[Llama Cpp Code Diff]] | Code-level diff of mainline llama.cpp, charlie12345/ROCmFPX, and ik_llama.cpp with a focus on quantization code, ggml… |
| [[Llama Cpp Runtimes]] | Compared mainline llama.cpp, charlie12345/ROCmFPX, and ik_llama.cpp — positioning, quantization formats, code-level… |
| [[Local TTS and Voice Cloning]] | How to create a new local TTS voice from sample files in oh-my-pi, the 2026 open-weight voice-cloning landscape, the… |
| [[Most Popular Self-Hosted AI Apps]] | The 15 most popular self-hosted AI applications of June–September 2026, ranked by GitHub stars, with homelab fit and… |
| [[PICO-8 Asset Standards]] | Production-grade standards for PICO-8 asset sheets — palette discipline, single perspective and light direction,… |
| [[Service Research Brief]] | Consolidated research from the `frank-zealot` research subagents (2026-08-14) plus the pi agent knowledge base. |
| [[SGLang]] | What SGLang is, how it works (RadixAttention, paged KV, overlap scheduler, structured output), how it compares to… |
**Operations:**
## LLM technique and hardware references
- [[Remote Job Runner]] — detached long-job execution with sentinel files
- [[Payload and Disk Reclaim]] — disk reclaim: measure, classify, prove,
stage, ask; plus purge-and-refresh
- [[Container Host Reachability]] — netavark/pasta reachability quirks and
the pasta port gate
- [[Fleet Ops Playbook]] — endpoint-down GPU-holder reclaim, scratch-boot
stop-first, pre-commit iteration, incident-window reconstruction, WoL
- [[Fleet Audit and Posture]] — monitoring gaps, vhost probes, security
posture, whole-fleet audit, bulk redaction
- [[SSH and Remote Access]] — passwordless bootstrap, fish login shells,
GUI-hang diagnosis over ssh
| Page | Owns |
|---|---|
| [[Dankclaw Improvements]] | This page collects concrete, sourced recommendations to improve Dankclaw |
| [[Dankclaw System Specs]] | Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference. |
| [[dankclaw-iommu-translation-off]] | Dankclaw boots iommu=pt since 2026-10-18 — identity DMA with the IOMMU armed, which restored the XDNA2 NPU at a… |
| [[Dankserver System Specs]] | Dankserver hardware and OS state since the 2026-09-20 board swap (ASUS ProArt B850-Creator WiFi NEO, 2x AMD Radeon AI… |
| [[dankserver-frontier-moe-feasibility]] | VERIFIED 2026-09-28 by measurement and source reading. Compares DeepSeek-V4.1-Flash, MiMo-V2.6-Flash-MOPD and… |
| [[dankserver-vram-growth]] | Investigation of the recurring cuBLAS crash on frank-zealot. What grows, what does not, and what to do next. |
| [[dealign-ornith-crack]] | dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with… |
| [[deployment]] | Scoped deployment and update process for the three inference services (frank-archon, frank-dragoon, frank-zealot). |
| [[DFlash DSpark Speculative Decoding]] | DFlash and DSpark are the two newest speculative decoding methods. |
| [[Dual-R9700-Cooperation]] | Why frank-medium's two Radeon AI PRO R9700s could not run tensor-parallel on the old B650-E board (PCIe atomic… |
| [[FastFlowLM NPU Inference]] | FastFlowLM (FLM), the AMD NPU-first LLM runtime for Ryzen AI XDNA2 chips. Fit analysis for the 50 TOPS NPU on… |
| [[FreeToken Builds]] | Building FreeToken (the official franksdank fork of FreeTok Tokenworks' freetoken vLLM-MTP runtime) on Dankclaw — the… |
| [[freetoken-benchmark-sweep]] | KV pool vs decode speed sweep across bs1/bs3/bs4/bs6 on Dankserver with Qwen3.6-35B-A3B-FP8 via FreeToken v0.1.2. |
| [[GGUF Compatibility]] | How to answer "will engine X load this GGUF?" from the file's header alone, why a custom tensor type forces the… |
| [[Heterogeneous Split Triage]] | Triage before building when someone asks to split a model across unequal accelerators on one host (24 GB GDDR6X +… |
| [[Hybrid CPU GPU Processing Dankserver]] | Dankserver uses NVIDIA GPU with system RAM for hybrid processing. |
| [[ling-3.0-flash-trial]] | `bailingmoe3-support` (commit `db4480bc8`, build b10411) at `/home/dank/llama.cpp-ling`. |
| [[ling-3.0-flash-vllm-plan]] | This page was the plan for a vLLM-ROCm deployment of `jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native` (INT4 W4A16, ~62… |
| [[Llama Cpp Multi-Device Splits]] | What llama.cpp can and cannot express across devices, the no-collective layer-split path, split balancing by… |
| [[Llama Cpp Serving Sizing]] | Slot-pool sizing against a headroom band, per-device VRAM accounting, prompt-cache/state sizing and the QSA… |
| [[LLM Bench Windows and Measurement]] | Stopping-and-restoring benchmark windows on the fleet LLM hosts (llama-bench sweeps, HTTP param benches, multi-arm… |
| [[LLM Performance Diagnosis]] | Probing why a served LLM is slow — the flat-depth probe, the expert-offload duty-cycle arithmetic, kernel-time… |
| [[Model Swap and Rollback]] | The full model-swap discipline for the LLM fleet — fit preflight, fetch and payload verification, runtime build/flag… |
| [[PCIe Link and Atomics]] | Link probing (idle downshift, path ceilings, CUDA/HIP bandwidth probes), the streaming-vs-compute ceiling arithmetic,… |
| [[Quantization by Hardware Architecture]] | This page maps LLM quantization formats to the GPU and CPU architectures that support them. |
| [[Quantization Production]] | — |
| [[qwen3.8-strix-optimizations]] | This page answers one question: **how do we make Qwen3.8-27B serve faster on |
| [[RAM CPU Offloading in VLLM]] | vLLM uses CPU RAM and swap space to extend GPU memory capacity. |
| [[RDNA4 Container Serving]] | Bringing a model larger than VRAM onto Dankserver's 2x R9700 (gfx1201) with the tcclaviger vLLM stack — quant-class… |
| [[RDNA4 MoE Expert Offload VRAM Budget]] | The expert-offload plan arithmetic on the tcclaviger fork — which terms a deploy can move, the util law, the… |
| [[Running Qwen3.6 27B FP8 on Dankclaw]] | Qwen3.6-27B is a dense vision-language model from the Qwen3.6 series. |
| [[Running Qwen3.6 35B A3B FP8 on Dankserver]] | Qwen3.6-35B-A3B is a Mixture-of-Experts vision-language model from the Qwen3.6 series. |
| [[Shared Memory Architecture Dankclaw]] | Dankclaw uses AMD Strix Halo APU with unified memory architecture. |
| [[swift15-campaign-worklog]] | — |
| [[Tcclaviger Fork Ops]] | Operating the tcclaviger RDNA4 vLLM fork on dankserver (shape G era, rollback target since the 2026-10-05 FreeToken… |
| [[ubatch-benchmark-large]] | Standardized ubatch sweep for the active kingjones Qwen3.8 ROCmFPX deployment. |
| [[vLLM Boot Forensics and Config Interrogation]] | Capturing a vLLM container boot failure's real root cause from journald (the container log is already gone),… |
| [[VLLM Index]] | This index contains all documentation for vLLM deployment on Dankserver and Dankclaw. |
| [[VLLM Overview]] | vLLM is a fast and easy library for large language model inference and serving. |
| [[vLLM Porting and Compile Triage]] | Forward-porting a model architecture to a new vLLM major (additive-only discipline, gap enumeration, shims that… |
| [[vllm_research_findings]] | vLLM v0.26.0 (latest stable, released July 25, 2026) supports DFlash speculative decoding with the `--spec-method… |
| [[x4-pair-experiment]] | What mainline llama.cpp can and cannot do with a 3090 + Strix Halo combination, measured 2026-09-16 on Dankclaw — the… |
**Architecture:**
## Pipeline Parallelism (series)
- [[Stale Signal Audit]] — classifying green-but-dead and down-but-fine
signals
- [[Quadlet Lifecycle]] — quadlet units, drift, crash-loops, compose
cutover
- [[Deploy State Triage]] — revision markers, pending lockouts, deploy
interruptions, shared-checkout hygiene
| Page | Owns |
|---|---|
| [[Ethernet Bandwidth Math]] | Bytes-per-token math for pipeline vs tensor parallelism, feasibility tables for 1/2.5/10 GbE, and real multi-node… |
| [[Expected Performance]] | Estimated tokens per second for a 25-40B model split across two nodes over 1/2.5/10 GbE, with real benchmark anchors. |
| [[llama.cpp RPC]] | llama.cpp distributed inference via ggml-rpc-server and the --rpc flag: architecture, CLI, versions, limitations, and… |
| [[Mixed-Vendor Experiments]] | Real-world NVIDIA + AMD Strix Halo multi-node experiments: what works (llama.cpp RPC, Vulkan split), what breaks… |
| [[Qwen3.6 27B Spec Methods]] | Qwen3.6-27B Q8 GGUF variants and llama.cpp speculative decoding (draft-mtp, draft-dflash, ngram-mod): flags, RPC… |
| [[Qwen3.8 PP Deployment]] | Split Qwen3.8-27B across the Strix Halo iGPU (head) and the RTX |
| [[Sample Script Set]] | An educational two-node llama.cpp RPC split of Qwen3.6-27B-Q8_K_P over ethernet. For reading only. Do not run or… |
| [[vLLM Pipeline Parallelism]] | vLLM multi-node pipeline parallelism: flags, Ray/mp backends, interconnect requirements, PP bubble, and PP +… |
**Infrastructure:**
## Uncategorised
- [[Observability Provisioning]] — Prometheus/Grafana config as code
- [[Homepage Status Probes]] — the status page's checker, vantage rules,
and triage
**LLM cross-cutting (full index: [LLM/index.md](LLM/index.md)):**
- [[GGUF Compatibility]] · [[Llama Cpp Serving Sizing]] ·
[[Llama Cpp Multi-Device Splits]] · [[PCIe Link and Atomics]] ·
[[RDNA4 Container Serving]] · [[RDNA4 MoE Expert Offload VRAM Budget]] ·
[[vLLM Boot Forensics and Config Interrogation]] ·
[[vLLM Porting and Compile Triage]] · [[LLM Bench Windows and Measurement]] ·
[[LLM Performance Diagnosis]] · [[Model Swap and Rollback]] ·
[[Tcclaviger Fork Ops]] · [[Heterogeneous Split Triage]] ·
[[FreeToken Builds]] · [[Quantization Production]]
**Agent:**
- [[OMP Harness]] — compaction arithmetic, per-model samplers, preamble
audits
- [[Agent Docs and Edits]] — doc-structure audits, hashline edit recovery
- [[Command Center Dotfiles]] — the ~/.omp repo-as-deployment
**Desktop / Research:**
- [[Noctalia and Niri]] — the desktop shell, palette recolor, template
rendering, rotation pinning, iCX3 RGB
- [[PICO-8 Asset Standards]] — cohesive PICO-8 asset sheets
## LLM Fleet
The LLM inference fleet on the two hosts. Full index:
[serverAppConfigs-docs/LLM/index.md](LLM/index.md).
### Deployment and Topology
- [[deployment]] — unified deploy and update process for both servers
- [[frank-archon-radiance-qwen38-iq4r]] — **ACTIVE frank-archon runbook
(2026-10-15)**: the PUBLISHER's container of the official
Qwen3.8-Flash-Next (`qwen3.8-next-flash-fp8-iq4r-moe.rad`, 116,322 MiB,
w4nl 4-bit-codebook experts + int8 trunk, Hub-sha256-verified at fetch)
swapped in as a MODEL_FILE-only change — the 2026-10-13 shape (slab
25600 / fp8-KV 5120, 3 × 200,000, host pool 44,928, MTP depth 2) carries
verbatim; rollback = revert the swap commit (the w4dn8 `.rad` stays on
disk; the fp8-native `.rad` was deleted 2026-10-15, rebuildable)
- [[frank-archon-radiance-swift15-w4dn8]] — RETIRED 2026-10-08 → 10-15 frank-archon runbook:
Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 `.rad` (radiance
1.0.13), gated on KLD ≤ 0.03 vs the BF16 container and decode ≥ 30.8 t/s;
activates by flipping `MODEL_FILE` in `scripts/frank-archon.env`, rollback
= revert that line (the FP8-native `.rad` stays on disk)
- [[frank-archon-radiance-qwen38-native-fp8]] — the RETIRED 2026-10-05 → 10-08 frank-archon
runbook (2026-10-05): the StillDeadcode radiance engine (digest-pinned
287 MiB image, libr4d gfx1201 kernels) serving the OFFICIAL
`Qwen/Qwen3.8-Flash-Next-FP8` at native block-FP8 (no-recipe convert
after the descriptor-staging drop) on BOTH R9700s TP=2 `wht6`, experts
in a 40 GiB/rank pinned host pool, PLE 47.7 GiB pinned to RAM, fp8 KV
7.32 KiB/token, MTP depth 3, 2 × 200,000 first boot; rollback is the
FreeToken one-card era (tag `frank-archon-freetoken-pr587-20261005`),
then the tcclaviger shape-G era (tag `frank-archon-vllm-shapeG-20261003`)
- [[swift15-campaign-worklog]] — ACTIVE campaign log: the Swift1.5 →
w4dn8 radiance convert and frank-archon swap in flight (branch
`swift15-radiance-w4dn8`); the cutover runbook replaces it at completion.
- [[frank-archon-freetoken-pr587]] — RETIRED 2026-10-05 (superseded the
same day by the radiance cutover above; kept as the rollback runbook):
FreeToken built at PR #587 head `77a7201` on the host into
`frank-archon-freetoken:pr587-77a7201`, ONE R9700 with experts in pinned
host RAM behind a 0.10 GPU LRU, 2 slots, parsers AUTO
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
- [[frank-archon-r9v-radiance-qwen38-q8]] — RETIRED 2026-09-30 (two swaps
back; rollback target): the R9V radiance-vLLM fork (docker-pinned bundle
`2dac17a2`, docker→podman shim) serving the unsloth
Qwen3.8-Flash-Next-GGUF Q8_0 (175.30 GiB, 6 shards) + blockfp8 MTP4 +
BF16 vision on Dankserver's both R9700s — TP=2 tiered experts (80-slot
LRU cache), PLE pinned 54.4 GB in host RAM, CED off, 131,072 ctx single
stream, Qwen-Sharp template, root unit with LimitMEMLOCK, the on-host
qualified state, the cutover, and the rollback
- [[frank-archon-r9v-radiance-qwen38-uncensored]] — RETIRED 2026-09-27
(rollback target): the R9V radiance-vLLM fork serving the abliterated
Dyluhn Qwen3.8-Flash-Next-Uncensored IQ4_XS (92.39 GiB) + MTP4 + F16
vision — TP=2 tiered experts with a UVA pinned host copy, PLE on
ssd-mmap, CED on, 196,608 ctx; state dir + frozen config + payload
intact on disk
- [[frank-archon-llamacpp-qwen38-flash-next-q8]] — RETIRED 2026-09-27
(rollback target): unsloth llama.cpp (b11030-mix prebuilt, ROCm
gfx120X, root unit) serving unsloth Qwen3.8-Flash-Next at Q8_0 (175.3
GiB) on Dankserver's both R9700s — the shared Q8_0 MTP head chained
behind ngram-mod, BF16 vision, the experts and the 51B n-gram table
computed in host RAM, the placement law, the cutover, and the rollback
- [[frank-archon-lucebox-ds4]] — RETIRED 2026-09-26 (same-day swap;
rollback target): the Lucebox runtime serving
DeepSeek-V4.1-Flash-ROCMFP23 (377.2 GiB GGUF, **40**-layer MoE, **384**
routed experts top-6, 189 GiB of embedded Engram) on Dankserver's R9700
with the routed experts streamed from the mmapped page cache — held :7770
for hours
- [[dankserver-frontier-moe-feasibility]] — VERIFIED 2026-09-28: can
DeepSeek-V4.1-Flash (552B), MiMo-V2.6-Flash-MOPD (309B) or GLM-5.3-Flash
(320B) run fast on the 2x R9700 box, and can Affinity / vllm-radiance /
FreeToken-AMD be modified for them? The capacity arithmetic (475 GiB
payload, 189 GiB of it Engram, ~2.4 GB of cold expert weights per token
across PCIe = a ~23 tok/s ceiling) plus the Affinity V4-Flash-to-V4.1 port
assessment — reusable expert tiering, four architecture gaps, one a
silent-wrongness trap on the Engram key
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — VERIFIED 2026-09-29:
the flag-level decode-tuning campaign on the ad hoc qwen-fp8 container
(tcclaviger RDNA4 fork, orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8,
TP=2 across both R9700s). **>35 tok/s MET: 37.44-40.57 tok/s with
`--expert-prefetch` DELETED** — telemetry measured the prefetcher at
36% precision, wasting ~half the gather traffic (the LRU itself runs at
82.45% hit); the prefetch-on incumbent 26.8 survived every other
challenger (geometry, spec depth, TunableOp-corrupts, batch knobs,
concurrency, ngram-fork-bug); the residual gap to ~60 tok/s is the
fork's gather read path (6.5-9.5 GB/s effective vs 22.7 at the same
slab size)
- [[frank-archon-glm53-flash-w4a16]] — VERIFIED 2026-09-30: the ACTIVE
frank-archon runbook. The cutover to Intel/GLM-5.3-Flash-W4A16-AutoRound
(symmetric int4 g128) on the tcclaviger fork, and the reason the int4
AutoRound payload is the ONLY one that serves here (its `auto-round` quant
method routes to the W4A16 offload kernel; a compressed-tensors W4A16
checkpoint of the same model reaches TritonWNA16Experts, which has no
offload recipe). **Measured decode 9.9-10.0 tok/s / prefill 918 tok/s; the
35 tok/s target is NOT met** — the step is bounded by the fork's gather
read path (~8.2 GB/s vs a 22.74 GB/s link) with the cache at a 0.53-0.66
hit rate, and MTP4 was measured and rejected (zero decode gain, prefill
918 -> 255). Carries the fork's KV-cap bug, the fp8-KV/gfx12 trap, and the
`glm47` tool/reasoning parser wiring
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-26:
vllm-radiance as ONE tensor-parallel process (TP=2 over the R4D one-shot
allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4
W4A8, DFlash2 drafter, unquantized bf16 KV since 2026-09-21) on Dankserver's
2x Radeon AI PRO R9700 behind the :7770 endpoint — the quantization-aware
entrypoint, the three-arm battery that chose TP=2, verified capabilities,
and the record of the 2026-09-16 two-replica era
- [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium:
the streamlining toolchain (hf-manifest.py, hf-fetch.sh,
swap-frank-medium.sh), the quantization-aware entrypoint contract, the
quadlet edit, the deploy, the capability battery, and the rollback path
- [[frank-medium-llamacpp-dflash2]] — RETIRED 2026-09-16 → 2026-09-20:
mainline llama.cpp (gfx1201 HIP build) serving orcarouter Qwen3.8-27B at
Q8_0 with the DFlash2 drafter as ONE layer-split process (32:33, two
163,840-token slots); holds the batch/ubatch optimum curve and the
per-layer split arithmetic the rollback recipe points back to
- [[Dual-R9700-Cooperation]] — whether the two R9700s can run ONE model: the
RCCL/PCIe-atomics evidence on the old TUF GAMING B650-E (which closed tensor
parallelism too), what the 2026-09-20 ProArt B850-Creator x8/x8 swap changed
— both chains `Routing+`, both endpoints `ReqEn+` — and the measured
three-way verdict: radiance TP=2 won and is the deployed shape
- [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era):
charlie12345/ROCmFPX serving kingjones777/
Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the runtime it
replaced
- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep (bs1/bs3/bs4/bs6, 8k–700k KV) on the retired Dankserver RTX 3090
- [[kingjones-strix-mtp]] — the Qwen3.8-27B-Q6_0 ROCmFPX AGENT era with the
standalone MTP draft (Dankclaw, 2026-08-25 → 2026-08-29; historical)
- [[agentionai-fast-imatrix]] — the retired frank-dragoon runbook:
agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87.06 GiB,
Q4_0_ROCMFP4_FAST) + own MTP draft, build-qwen38 (retired 2026-08-30;
rollback record)
- [[frank-dragoon-halogen27b-dflash2]] — RETIRED 2026-10-05 → 10-06 frank-dragoon
runbook (root systemd unit + podman, port 7770): the digest-pinned
ghcr.io/peonist-ai/halogen image serving peonist-ai/halogen-qwen3.8-27b
(qwen3.8-27b-p1w4d-d2.hgn, 33.4 GiB — a .hgn that loads in no other
engine) with the DFlash2 drafter on ONE slot of 200,700 ctx (slots>1
disables speculation — engine law); TEXT-ONLY and censored base, both
accepted tradeoffs; no server-side sampler, so the thinking recipe rides
the wire
- [[frank-dragoon-flash-server-v2-native]] — **ACTIVE frank-dragoon runbook
since 2026-10-17**: halogen-flash-server 0.16.2 serving the **NATIVE v2
abliterated** conversion of orcarouter's pinned BF16
(`qwen38-flash-next-v2-abliterated.hgn`, 62.1 GiB) at ctx **200,704** over a
**401,408**-position pool (`HALOGEN_KV_POOL_POSITIONS=401408`, 11.0 GiB
reserved — one pool behind both slots, raised to 2 × ctx and deployed
2026-10-18 so TWO maximum-length
requests coexist) and the stock peonist n-gram/vision companions **hardlinked**
in — engine-tool certified (NIAH 18/18, PPL 3.968, mean KL 0.155); MTP
always-on, vision ON, no server-side sampler change
- [[frank-dragoon-flash-server-v2-swift15]] — RETIRED 2026-10-06 → 10-17
frank-dragoon runbook (rollback target — payload + fetch script on disk):
halogen-flash-server 0.16.2 serving the abliterated **Swift1.5 finetune**
of the Qwen3.8-Flash-Next v2 `.hgn`, two slots over a 524,288-position
pool; owns the 400-zone law and the sampler/prompt-cache contract
- [[frank-dragoon-rocmfpx-q6agent-uncensored-mtp]] — RETIRED 2026-10-03 → 10-05
(rollback target; payload, fork build and build script on disk): the pinned
kingjones30/ROCmFPX fork (scripts/build-rocmfpx.sh) serving kingjones777/
Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF
**Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT** — ftype 114, 339 tensors of
GGML_TYPE_Q6_0_ROCMFPX + 159 type-103, a format no other loader on the
host reads — with the repo's Q4_0 MTP head drafting 4 tokens (27.6-28.1
t/s measured, 86.8 % acceptance; the ngram-mod chain measured and
rejected) and its BF16 projector, one session at the native 262,144 window
- [[frank-dragoon-rocmfpx-q8agent-mtp]] — RETIRED 2026-10-02 → 10-03 (rollback
target, aligned Q8 payload still on disk): the same fork serving the
ALIGNED Q8_0_ROCMFPX_AGENT at 17.0 t/s — the +123 % MTP A/B, the traps
- [[frank-dragoon-gufo-qwen38-27b]] — RETIRED 2026-10-01 → 10-02 (history;
the payload dirs were removed from disk 2026-10-03, `scripts/build-gufo.sh`
remains): gufo v0.1.1
built from source serving Qwen3.8-27B UD-Q4_K_XL + the z-lab DFlash2
drafter on the gfx1151 iGPU — the two from-source build workarounds, the
template-hash allowlist, the measured prefill/decode trade
- [[frank-dragoon-vllm-mimo-v26-bf16]] — RETIRED 2026-10-01 (same day;
replaced by the gufo rollback, then by
[[frank-dragoon-rocmfpx-q8agent-mtp]]): the digest-pinned
kyuz0/vllm-therock-gfx1151 image (vLLM 0.30, TheRock ROCm 10) serving
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B in FULL BF16 on the gfx1151 iGPU,
4 x 262,144 ctx, with CROSS-MODEL MTP spec decode from Qwen/Qwen3.5-9B —
the roof math (13.1 t/s base is ~94% of LPDDR5x BF16), the measured
MTP-vs-DFlash table (19.8 vs 17.4 t/s), the greedy-text numerics trap,
the `reasoning` field change, troubleshooting
- [[frank-dragoon-unsloth-qwen38-27b-q8xl]] — RETIRED 2026-09-28 (replaced by
[[frank-dragoon-gufo-qwen38-27b]]; rollback target): stock llama.cpp
(unsloth b11030, ROCm gfx1151) serving
unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL with the z-lab DFlash2 Q4_K_M drafter
(depth 4) on the gfx1151 iGPU — one 262,144-token conversation, q8_0 KV
(the build lacks fp8), text-only, Qwen-Sharp v22.4.0 template, the
114 GiB GTT memory model, client contract
- [[frank-dragoon-kairic-edge-qwen38-27b]] — RETIRED 2026-09-21 (replaced by
[[frank-dragoon-unsloth-qwen38-27b-q8xl]]; same-day swap): the Kairic Edge
runtime (ciru-ai/ROCmFPX kairic-edge-qwen38-27b-v1.2, TheRock 7.15.0,
Composable Kernel) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (a 27B
dense model, main GGUF + FFN/GDN/GDN-Output .pfs sidecars) on the gfx1151
iGPU — one 262,144-token conversation, native MTP depth 4 + n-gram
drafting, compatibility mode, the 114 GiB GTT memory model, client contract
- [[frank-dragoon-unsloth-qwen38-flash-next]] — RETIRED 2026-09-21 (replaced by
[[frank-dragoon-kairic-edge-qwen38-27b]]; same-day swap): the unsloth
llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving
unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (a 125B MoE / 6B active) on the
gfx1151 iGPU — one 200,000-token conversation, MTP depth 2 via the
explicit -md shared-Q8_0 head, the 114 GiB GTT memory model, client contract
- [[frank-dragoon-ciru-orca]] — RETIRED 2026-09-21 (replaced by
[[frank-dragoon-unsloth-qwen38-flash-next]]; rollback target, 2026-09-19 →
09-21): the CIRU v4.4.1 runtime (private host build, QSA hotfix) serving
jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 iGPU — one
262,144-token conversation, MTP depth 4, 4096 MiB PLE cache, runtime-root
gate, client contract
- [[halogen-flash-server]] — RETIRED 2026-09-19 (replaced by
[[frank-dragoon-ciru-orca]]; rollback target, 2026-09-16 → 09-19):
peonist-ai/halogen-flash-server 0.5.8 serving peonist-ai/halogen-qwen3.8-
flash-next (w4b `.hgn` + quality overlay) on the gfx1151 iGPU — two 200k
conversations over one 400,384-position pool, deployed flags, memory model,
client contract
- [[frank-dragoon-ik-uncensored]] — the RETIRED 2026-09-16 frank-dragoon runbook
and rollback target (2026-09-15 → 09-16): ik_llama.cpp serving
mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090 +
Strix Halo hybrid — its measured envelope is why the fleet went back
- [[dankclaw-iommu-translation-off]] — Dankclaw's IOMMU boot line: `iommu=pt`
since 2026-10-18 (measured cost of identity DMA: nothing — +2.0 % prefill), the
`amd_iommu=off` era it replaced, and the tracked file + installer that own it
- [[kingjones-strix-rocmfp4]] — the PRIOR frank-dragoon runbook
(2026-08-30 → 2026-09-11) and the rollback target:
kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF on the card's
pinned kingjones30/ROCmFPX build
- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on Dankclaw
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
- [[frank-zealot-llama-prism-bonsai9-mtp]] — the **ACTIVE** frank-zealot
runbook since 2026-10-18: the PrismML vendor llama.cpp build 10777 (the only
loader for the custom PQ2_0 tensor type / `prism.hadamard.*`, and the floor
for MTP beside a quantized V cache) serving
prism-ml/Ternary-Bonsai-2-9B-gguf **PQ2_0 with the MTP head** on the RTX 3090
— TWO 262,144-token conversations (`--ctx-size 524288 --parallel 2`), exact
f16 K + f16 V = **16,384 MiB** over 8 full-attention layers (32,768 B/token;
the per-slot window is UNCHANGED at 262,144 — what halved is the POOL,
because exact f16 over the OLD 786,432 pool bills **24,576 MiB** against the
**24,123** the driver leaves addressable; the old "26,112 vs 13,056 MiB" note
doubled the q8_0 figure with its 1/32 scale overhead included, so f16/q8_0 is
1.882×, not 2×), MTP **off by measurement** (the draft context bills 2,048
MiB of f16 KV over the whole pool, and exact KV leaves no room for one),
vision through the Q8_0 mmproj; boot 17,876 used / 6,249 free (the q8_0 era)
- [[frank-zealot-llama-prism-bonsai2-pq20]] — RETIRED 2026-10-17 → 10-18
(rollback target, payload + build 10754 on disk): Ternary-Bonsai-2-**27B**
PQ2_0 at TWO 200,704-token conversations, q8_0 K + q8_0 V, MTP **impossible**
(no head in that export), vision through the Q8_0 mmproj; boot 22,664 / 1,461
- [[frank-zealot-llama-qwen35-9b-q6k-2x200704]] — RETIRED 2026-10-17 (rollback
target, payload on disk): llama.cpp master (CUDA sm_86) serving
HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** on the RTX 3090 —
TWO 200,704-token conversations, exact f16 K + f16 V, MTP **impossible**
(the file carries no `nextn`/`mtp` tensors), vision through the publisher's
BF16 mmproj; priced from the MiMo Q6_K boots (same qwen35 geometry)
- [[frank-zealot-llama-gemma4-q4km-200704]] — RETIRED 2026-10-07 (same day;
payload on disk as rollback): llama.cpp master (CUDA sm_86) serving
HauhauCS Gemma4-26B-A4B QAT Uncensored **Q4_K_M** on the RTX 3090 — ONE
200,704-token conversation, f16 K + q8_0 V, MTP **disabled** (the
`gemma4-assistant` drafter costs ~1.9 GiB: boot was 23,806 used/319 free
with it, 21,890/2,235 without), vision through the publisher's mmproj
- [[frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688]] — RETIRED 2026-10-07
(superseded by gemma4; payload on disk as the rollback): the
frank-zealot runbook 2026-10-05 (evening) → 2026-10-07
(**UNCECERSED RCO-IQ3_XXS
with MTP back on since 2026-10-06 evening; renamed from frank-small and
reshaped to TWO 114,688-token conversations at Frank's 112k cap with
the ubatch ladder climbed to 2048 late 2026-10-06**): llama.cpp master (CUDA
sm_86) serving ajgazin's Swift-1.5-Qwen3.8-27B Uncensored RCO-IQ3_XXS
(the MTP head in the file is drafted) on the RTX 3090 — 1,005 MiB free
at boot (763 settled after a vision probe), q8_0 K + q8_0 V, vision
through the publisher's own projector, ~52–58 t/s recipe decode with MTP
- [[frank-zealot-llama-mimo-q6k-2x200704]] — RETIRED 2026-10-05 (evening,
superseded by swift15): llama.cpp master serving
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** — TWO conversations at
200,704 tokens each with fully unquantized f16 KV and mmproj vision;
94.4 tok/s decode (the Q6_K rung costs a measured −10 % vs Q5_K_M —
weight-bandwidth), batch 4096 / ubatch 2048 (+4.6 % prefill, measured
2026-10-05), 2,279 MiB free, and the ≥256 px image-probe trap (tiny
images answer "video")
- [[frank-zealot-llama-mimo-q5km]] — RETIRED 2026-10-02 → 10-03 (rollback
target, payload on disk; owns the measured boot-pair table): same runtime
and shape one rung lower — 104.8 tok/s, 3,571 MiB true free
- [[frank-zealot-llama-mimo-q6k]] — RETIRED 2026-10-01 (same day; that same
Q6_K rung returned ACTIVE 2026-10-03 at the new shape, see
[[frank-zealot-llama-mimo-q6k-2x200704]]): the first llama.cpp + GGUF shape
for this endpoint, Q6_K at 2 x 262,144 / f16-K + q8_0-V, 20.9 GiB measured
- [[frank-zealot-sglang-mimo-v26]] — RETIRED 2026-10-01 (rollback target):
SGLang in a self-contained python3.12 venv serving
letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound (qwen3_5
hybrid, INT8 W8A16, in-checkpoint vision, NO MTP weights shipped) on
Dankclaw's RTX 3090, port 7771 — fp8 KV, pinned 606,208-token pool,
2 x 262,144 slots, bf16 mamba states, eager prefill
- [[frank-zealot-gemma4-12b]] — RETIRED 2026-10-01 (rollback target):
mainline llama.cpp serving
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
(MTP assistant sidecar + ngram-mod, q8_0 KV) on
Dankclaw's RTX 3090, port 7771
- [[frank-zealot-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-
token pool, 2 concurrent full-context sessions, native text+image+audio
- [[frank-zealot-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target):
the prior frank-zealot runbook (2026-09-16 → 09-17), ISTA-DASLab
Qwen3.8-27B-GSQ-RCO IQ3_S with own in-file MTP head + BF16 projector
- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for the old frank-zealot (q8_0 + Hadamard KV cache, split-mode none; 2026-08-26)
- [[freetoken-frank-small]] — RETIRED 2026-09-15: the FreeToken/CUDA frank-small runbook, retired with the RTX 3090
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment (2026-08-22)
- [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment (2026-08-21)
- [[Dankserver System Specs]] — the 2x R9700 box hardware (frank-medium)
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-dragoon)
- [[Dankclaw Improvements]] — host-level changes on the Strix Halo box
- [[Swap Configuration]] — the 64 GiB `/swap.img` host swapfile (raised from
8 GiB 2026-09-12): fstab-driven `swap.img.swap` unit, the safe resize
ordering that drains the old file into fresh swap instead of into a host
whose `MemAvailable` lies by the size of the model, and the swappiness caveat
- [[frank-npu]] — the RETIRED FLM NPU endpoint on Dankclaw (2026-09-11):
Gemma4 E4B vision+audio pack on the XDNA2 NPU, port 7771; kept as the
retirement record, including the IOMMU/SVA err=-19 root cause and
where every dependent route moved
### Benchmarks and Sizing
- [[frank-dragoon-benchmark]] — backend/spec benchmark behind the old head
- [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results
- [[frank-zealot-benchmark]] — the small-model benchmark suite
- [[frank-medium-batch-ubatch]] — the 2026-09-16 batch/ubatch sweep on the 2x
R9700 layer split: prefill PEAKS at ubatch 512 and falls on both sides, the
logical batch buys nothing measurable, decode never moves, and the ubatch's
uneven VRAM give-back (1.30 GiB on card 0 against 2.41 GiB on card 1) moved
the tensor split from 36:29 to 35:30
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware notes
- [[Running Qwen3.6 27B FP8 on Dankclaw]] — Qwen3.6-27B sizing on the iGPU
- [[qwen3.8-strix-optimizations]] — Strix Halo tuning notes
- [[x4-pair-experiment]] — what a 3090 + Strix Halo combination can and cannot
do in mainline llama.cpp (measured 2026-09-16): the tensor inventory that
decides a split, the one flag (`-ot`) that kills per-tensor placement across
devices, the two-device layer splits that DO work (pure-Vulkan and cross-vendor
CUDA+Vulkan, +7 % decode at 64k), and the harness-parsing trap that made the
first round of measurements say otherwise
### Memory and Hardware
- [[RAM CPU Offloading in VLLM]] — KV cache swap between GPU VRAM and system RAM
- [[Hybrid CPU GPU Processing Dankserver]] — NVIDIA GPU compute with system RAM offload
- [[Shared Memory Architecture Dankclaw]] — AMD unified memory and ROCm configuration
- [[dankserver-vram-growth]] — VRAM/RAM growth behind the retired cuBLAS crash
- [[Quantization by Hardware Architecture]] — which quant format fits which GPU/CPU
### Retired Topology and History
- [[Qwen3.8 PP Deployment]] — the retired pipeline-parallel head + worker
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history (superseded by FastMTP)
- [[FastFlowLM NPU Inference]] — the retired FastFlowLM NPU endpoint
- [[DFlash DSpark Speculative Decoding]] — retired DFlash/DSpark drafting
- [[GGUF Compatibility]] — which loaders accept which quant formats; the tensor-type histogram check
- [[Llama Cpp Multi-Device Splits]] — layer vs tensor split across unlike devices; the -ot pipeline gate
- [[Llama Cpp Serving Sizing]] — VRAM arithmetic from two measured boots, never a formula
- [[PCIe Link and Atomics]] — link caps vs loaded rate; RCCL atomics routing on the old board
- [[RDNA4 Container Serving]] — the serving stack recipe: mounts, ulimits, udev, verify gates
- [[RDNA4 MoE Expert Offload VRAM Budget]] — arena sizing law for expert offload: free VRAM is estimator slack, not allocatable headroom
- [[vLLM Boot Forensics and Config Interrogation]] — journal/endpoint forensics for boot failures; argv recovery from a live engine
- [[vLLM Porting and Compile Triage]] — forward-porting vLLM majors; compile failure forensics
- [[ling-3.0-flash-trial]] — the Ling-3.0-Flash trial runbook
- [[ling-3.0-flash-vllm-plan]] — the Ling-3.0-Flash vLLM plan (retired)
### vLLM Reference (historical)
- [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
- [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
- [[vllm_research_findings]] — research findings from the vLLM era
### Pipeline Parallelism (archive)
- [[Ethernet Bandwidth Math]] — the bandwidth budget for the PP pair
- [[Expected Performance]] — expected PP throughput
- [[llama.cpp RPC]] — the RPC transport between head and worker
- [[Mixed-Vendor Experiments]] — cross-vendor PP experiments
- [[Qwen3.6 27B Spec Methods]] — spec-decoding methods for the 27B
- [[Sample Script Set]] — reference scripts for the PP pair
- [[vLLM Pipeline Parallelism]] — vLLM's PP implementation
## Update Log
- [[log]] — chronological record of knowledge-base changes
## See Also
- [AGENTS.md](../AGENTS.md) — repository rules
- [serverAppConfigs-docs/LLM/index.md](LLM/index.md) — the LLM fleet knowledge base
- [dankware Configuration Knowledgebase](~/.pi/agent/docs/index.md) — the "root" level system knowlegebase
| Page | Owns |
|---|---|
| [[log]] | — |