dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

Merge origin/main into docs/radiance-prefill-decode-census

Browse Source 
Two conflicts, both from the 2026-10-19 vault compaction on main:

* frontmatter description — main shortened it to satisfy OKF DESC
  (max 120 chars); this branch's longer wording would have failed the
  seam. Kept main's shape and folded the new §9 in by naming
  step-shape attribution instead of profiler attribution.
* the Related bullet for [[vLLM Porting and Compile Triage]] — that
  page was destroyed on main, so the link is dropped rather than
  resurrected; this branch's [[frank-archon]] row is kept.

Index rows regenerated with scripts/gen-vault-index.py.
This commit is contained in:
Frank
2026-10-19 14:20:05 -07:00
parent e3a02e25c7 9971cb559e
commit e133d2412d
164 changed files with 3435 additions and 15287 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/DFlash DSpark Speculative Decoding.md
+9 -13
View File
@@ -100,7 +100,7 @@ The DFlash drafter proposes, then ngram-mod runs as an additional implementation
`--spec-type` takes a comma list, and the gate is `ctx_dft` in
`common/speculative.cpp` (`common_speculative_init_result`): **draftless**
types (the ngram family — simple, map-k, map-k4v, mod, cache) are added
unconditionally, while **context-backed** types (`draft-mtp`, `draft-dflash`,
unconditionally, while **context-backed** types (`draft-mtp`, `draft-dflash`
`draft-dspark`, `draft-eagle3`) all require a draft CONTEXT. Consequences:
- A chain may hold ONE context-backed drafter plus any number of ngram types.
@@ -201,7 +201,7 @@ Speculative decoding is lossless under rejection sampling: the target verifies e
On the mid-August 2026 Dankclaw shape `draft-dflash` was the benchmarked winner
at 4k — 20.2 t/s, against 16.9 with ngram layered on and 8.7 ngram-only; those
three figures are that retired llama.cpp era's property ([[frank-dragoon]] owns
the era and its shape). MTP is a viable alternative to test, not a bug fix: the parallel-stream releases investigated on 2026-08-13 were client-side aborts and a Dankserver crash, not speculative decoding behavior (see [[deployment]]).
the era and its shape). MTP is a viable alternative to test, not a bug fix: the parallel-stream releases investigated on 2026-08-13 were client-side aborts and a Dankserver crash, not speculative decoding behavior (see ).
## Draft Models
@@ -257,7 +257,7 @@ DSpark took `ROCm error: an illegal memory access` inside the shared draft
context with two concurrent 16k prompts; DFlash corrupted multi-slot into
premature EOS and was retired first. The timeline, the crash class, the
"concurrency is the trigger, not size" finding and the `PARALLEL=1` workaround
are on [[qwen3.8-spec-matrix]]; the shape they were measured on is
are on ; the shape they were measured on is
[[frank-dragoon]]'s mid-August 2026 era.
## A/B protocol on a live llama.cpp endpoint
@@ -303,7 +303,7 @@ Protocol:
Read the right counters — per request `timings.draft_n` / `draft_n_accepted` /
`predicted_per_second`; cumulative `/metrics`
`llamacpp:spec_decode_num_drafts_total` / `..._accepted_tokens_total`;
`llamacpp:spec_decode_num_drafts_total` / `..._accepted_tokens_total`
per-task `draft acceptance = ... (N accepted / M generated)`. `draft_n == 0`
while tokens flow = the proposer is NOT engaged, whatever the flags say. A
zero-draft config is a useful negative control: it proves the other config's
@@ -367,7 +367,7 @@ states, so a dense-27B DFlash2 is useless for a 512-expert MoE trunk.
**Step 4 — acceptance reality checks.** Treat 40–60 % as the honest band for a
matched pair; the arm that anchors the bottom of it on this fleet is the
depth-7 `draft-dflash` row of [[frank-medium-llamacpp-dflash2]]'s spec table
depth-7 `draft-dflash` row of 's spec table
(accepted share, rate and VRAM for every depth are in that table). Cheapest
proxy for `p` at temp 0 when both models are greedy and share a tokenizer:
greedy continuation agreement on identical prompts, two API calls, no weights
@@ -381,10 +381,10 @@ drafting.
**Step 5 — check the attachment surface BEFORE the arithmetic.** Closed
engines refuse external drafters by design: halogen-flash-server's
`DRAFTER_CAP` gates `dflash2` on the loaded checkpoint and an unavailable
name is a deliberate 400 (a silent downgrade would make comparisons lie);
name is a deliberate 400 (a silent downgrade would make comparisons lie)
there is no device routing in its loopback token protocol, so a drafter can
never sit on another card. Pure llama.cpp DOES route the draft:
`--spec-draft-device/-devd` (default follows `--device`), `-ngld`,
`--spec-draft-device/-devd` (default follows `--device`), `-ngld`
`-otd`, `-ctkd/-ctvd`, `--spec-draft-n-max` (default 3), `--spec-draft-p-min`.
One process sees only the devices its backends were built with — CUDA+Vulkan
is the combination that shows a CUDA card AND an AMD iGPU (HIP+CUDA cannot
@@ -411,20 +411,16 @@ measurement that would settle it.
- Speculative decoding helps low-QPS, memory-bound, latency-focused workloads. It helps less at high QPS.
- DFlash needs a non-causal attention backend; the ROCm flash_attn backend must be validated before production use.
- The old `--swap-space` flag is retired; the live CPU-KV-offload knob pair is owned by [[RAM CPU Offloading in VLLM]].
-The old `--swap-space` flag is retired; the live CPU-KV-offload knob pair is owned by .
- `--speculative-config` is the preferred CLI. Legacy `--spec-method`, `--spec-model`, and `--spec-tokens` flags still exist.
- Use `rejection_sample_method` default `strict` for lossless output.
## See Also
- [[Quantization by Hardware Architecture]]
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
- [[Running Qwen3.6 27B FP8 on Dankclaw]]
- [[Dankclaw System Specs]]
- [[Dankserver System Specs]]
- [[VLLM Overview]]
- [[deployment]]
- [[GGUF Compatibility]] — header parsing and custom tensor-type lock-in
- [[Llama Cpp Serving Sizing]] — slot-pool and drafter KV sizing arithmetic
- [[Llama Cpp Multi-Device Splits]] — where the drafter's KV actually lands
- where the drafter's KV actually lands
- [[frank-dragoon]] and [[frank-zealot]] — the single-GPU env drop-in A/B vehicle
serverAppConfigs-docs/LLM/Dankclaw Improvements.md
+7 -9
View File
@@ -30,8 +30,8 @@ taken in.
- **Backend**: that era's production was **Vulkan0**, not ROCm. On this box ROCm
wins prefill and Vulkan wins decode, and the decode gap widens with context:
the 64k A/B is [[qwen3.8-strix-optimizations]] §4, the 262k A/B is
[[qwen3.8-spec-matrix]] §High-Context Backend Comparison. The "HIP
the 64k A/B is §4, the 262k A/B is
§High-Context Backend Comparison. The "HIP
out-performs Vulkan for decode" claim did not hold on the 8060S.
- **Power profile**: the `/sys/class/drm/card0/device/power_profile` file
does not exist on this machine (checked 2026-08-15). GPU DPM is already
@@ -39,7 +39,7 @@ taken in.
- **KV cache**: production ran `q8_0` K **and** `q8_0` V (both quantized). What
`frank-zealot` runs now is its own Current shape — this row is not it.
- **Spec**: DFlash/DSpark are retired. That era's production was `--spec-default`
(ngram-mod) with draft-mtp first — timeline on [[qwen3.8-spec-matrix]]; the
(ngram-mod) with draft-mtp first — timeline on ; the
spec setting each endpoint runs today is on its own page.
- **TTM**: the `pages_limit` raise that removed the ~15 GiB ceiling is recorded
once on [[Dankclaw System Specs]] (Memory).
@@ -54,8 +54,8 @@ taken in.
That 2026-08 reading is the firmware-VGM era: the BIOS carved ~96 GiB as
"dedicated" iGPU VRAM, so the kernel saw only ~30 GiB (the same split is
recorded in [[ling-3.0-flash-trial]] §Why Vulkan + GTT; the 76.2 GiB row is an
era [[frank-dragoon]] load reading). Every live number on this box — `MemTotal`,
recorded in §Why Vulkan + GTT; the 76.2 GiB row is an
era [[frank-dragoon]] load reading). Every live number on this box — `MemTotal`
the ~114 GiB GTT aperture that is the real GPU budget, the 2.0 GiB dedicated
carve and the `ttm` page cap — belongs to [[Dankclaw System Specs]], re-read
2026-10-18.
@@ -125,9 +125,9 @@ All require root and a reboot for the modprobe change.
## Sources
- [[frank-dragoon]] — measured backend/spec numbers
- [[deployment]] — deploy and update process
- deploy and update process
- AMD ROCm RDNA3.5 system optimization guide (gfx1151 memory/BIOS)
- llama.cpp source: `ggml/CMakeLists.txt`, `ggml/src/ggml-hip/CMakeLists.txt`,
- llama.cpp source: `ggml/CMakeLists.txt`, `ggml/src/ggml-hip/CMakeLists.txt`
`common/arg.cpp`, `common/common.h`
- NotebookCheck Ryzen AI MAX+ 395 analysis
@@ -135,5 +135,3 @@ All require root and a reboot for the modprobe change.
- [[frank-dragoon]]
- [[DFlash DSpark Speculative Decoding]]
- [[deployment]]
- [[index]]
serverAppConfigs-docs/LLM/Dankclaw System Specs.md
+4 -6
View File
@@ -32,7 +32,7 @@ Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp
| Device ID | 0x1586 |
| Architecture | gfx1151 (Strix Halo) |
| GPU-Visible Memory | ~114 GiB GTT aperture (the budget; `mem_info_gtt_total` = 122,402,963,456 B read 2026-10-18; rocm-smi's smaller SVM carve-out is not the budget) |
| Physical RAM | 121 GiB visible to the OS (`free -g` total 121, read 2026-10-18) on a 128 GB part — 8 × 16 GB LPDDR5 ([[ling-3.0-flash-trial]] §hardware). The 64 GiB this page recorded as "corrected 2026-08-17" (from 32 GB) is stale |
| Physical RAM | 121 GiB visible to the OS (`free -g` total 121, read 2026-10-18) on a 128 GB part — 8 × 16 GB LPDDR5 ( §hardware). The 64 GiB this page recorded as "corrected 2026-08-17" (from 32 GB) is stale |
### NPU
@@ -44,9 +44,9 @@ Dankclaw is an AMD APU + RTX 3090 inference host. It ran CIRU ROCmFPX llama.cpp
| Driver | amdxdna 0.7.0, in-tree — bound since 2026-10-18 (`lspci -k -s c6:00.1` → `Kernel driver in use: amdxdna`; journal 2026-10-18 01:25:14 `Initialized amdxdna_accel_driver 0.7.0 for 0000:c6:00.1 on minor 0`) |
| Firmware | 1.1.2.65 (`/usr/lib/firmware/amdnpu/17f0_11/npu_7.sbin` → `npu.sbin.1.1.2.65.zst`, loaded at boot 2026-10-18 01:25:13) |
| Runtime | None installed. FastFlowLM 0.9.45 + XRT (installed 2026-08-16) was **retired 2026-09-11** with the `amd_iommu=off` decision — see [[dankclaw-iommu-translation-off]]; the package remains at `/home/dank/GitPackages/fastflowlm_0.9.45_ubuntu26.04_amd64.deb` |
| Status | Not in service — no NPU endpoint is deployed. The device itself is usable since the 2026-10-18 `iommu=pt` boot (`/dev/accel/accel0` present, `crw-rw---- root:render 261,0`); Gitea issue #3 tracks enablement. See [[FastFlowLM NPU Inference]] |
| Status | Not in service — no NPU endpoint is deployed. The device itself is usable since the 2026-10-18 `iommu=pt` boot (`/dev/accel/accel0` present, `crw-rw---- root:render 261,0`); Gitea issue #3 tracks enablement. See |
The NPU endpoint is retired (2026-09-11; [[frank-npu]]). The `frank-zealot` name now serves **Ternary-Bonsai-2-27B-PQ2_0** on the RTX 3090 via the llama.cpp PrismML fork (see [[frank-zealot]]); the Gemma4-12B QAT rung it once pointed at is retired history ([[frank-zealot]], 2026-09-17 → 10-01).
The NPU endpoint is retired (2026-09-11; ). The `frank-zealot` name now serves **Ternary-Bonsai-2-27B-PQ2_0** on the RTX 3090 via the llama.cpp PrismML fork (see ); the Gemma4-12B QAT rung it once pointed at is retired history (, 2026-09-17 → 10-01).
### ROCm
@@ -80,7 +80,7 @@ The NPU endpoint is retired (2026-09-11; [[frank-npu]]). The `frank-zealot` name
| Interface | Type | Speed | IP | Notes |
|-----------|------|-------|----|-------|
| eno1 | onboard | 2500 Mb/s (2.5 GbE) | 192.168.0.143 (DHCP) | Carried the RPC path to Dankserver during the 2026-08-17 pipeline-parallelism trial; no RPC listener runs today (`ss -ltn` shows no :50052, 2026-10-18). Default route via 192.168.0.1. |
| enx00e04cc2e1e5 | USB-C to Ethernet adapter | 1000 Mb/s | 192.168.0.116 | UNPLUGGED 2026-08-17. While present, the route to 192.168.0.235 went through this adapter (lower route metric), so all RPC traffic was capped at 1 GbE. See [[Pipeline Parallelism/Ethernet Bandwidth Math]] section 0. |
| enx00e04cc2e1e5 | USB-C to Ethernet adapter | 1000 Mb/s | 192.168.0.116 | UNPLUGGED 2026-08-17. While present, the route to 192.168.0.235 went through this adapter (lower route metric), so all RPC traffic was capped at 1 GbE. See section 0. |
| docker0 | bridge | - | 172.17.0.1 | Usually down. |
The L2 gateway/switch at 192.168.0.1 (MAC f0:4b:8a:65:c5:40) is at
@@ -112,5 +112,3 @@ cable between claw and server is required for 10G.
- [[frank-dragoon]]
- [[Shared Memory Architecture Dankclaw]]
- [[frank-redo-2026-08-17]]
- [[FastFlowLM NPU Inference]]
serverAppConfigs-docs/LLM/Dankserver System Specs.md
+10 -11
View File
@@ -1,7 +1,7 @@
---
type: configuration
type: reference
title: Dankserver System Specs
description: Dankserver hardware and OS state since the 2026-09-20 board swap (ASUS ProArt B850-Creator WiFi NEO, 2x AMD Radeon AI PRO R9700 gfx1201 on CPU lanes x8/x8) serving frank-medium.
description: "Dankserver hardware and OS state since the board swap: board, lane topology, cards, memory, kernel posture."
tags:
- dankserver
- hardware
@@ -30,7 +30,7 @@ read 2026-10-18).
| Property | Value |
|----------|-------|
| Board | **ASUS ProArt B850-Creator WiFi NEO** (Rev 1.xx) — swapped in 2026-09-20 |
| Previous | ASUS TUF GAMING B650-E WIFI (card #2 was behind the chipset there; see [[Dual-R9700-Cooperation]]) |
| Previous | ASUS TUF GAMING B650-E WIFI (card #2 was behind the chipset there; see ) |
| GPU lanes | CPU GPP ports bifurcated **x8/x8** at Gen5: `PCIEX16(G5)_1` → `00:01.1`, `PCIEX16(G5)_2` → `00:01.3` (both bridges: `current_link_width=8`, `current_link_speed=32.0 GT/s`, read 2026-10-18); ~31.5 GB/s per card. Each card sits behind an on-board PCIe switch (`01:00.0`/`02:00.0`, `05:00.0`/`06:00.0` — "Navi 10 XL Upstream/Downstream Port of PCI Express Switch"), so the GPU endpoint itself trains **x16** Gen5 while the CPU-facing leg stays x8 |
| Atomic routing | `AtomicOpsCap: Routing+` on every bridge in both GPU chains, `AtomicOpsCtl: ReqEn+` on both endpoints — the condition that ruled out TP=2 for the previous board (2026-09-20 census; not re-readable today — `lspci -vv` needs root) *(unverified 2026-10-18)* |
| Free | `PCIEX16(G4)` (chipset, x4), M.2_2, M.2_3 |
@@ -55,7 +55,7 @@ read 2026-10-18).
| Architecture | gfx1201 (RDNA4) |
| VRAM | 32,624 MiB each (`rocm-smi` reports 34,208,743,424 B per card) |
| Link | PCIe 5.0, endpoint-trained **x16** each behind the on-board switch; the CPU-facing leg is x8 (`00:01.1` / `00:01.3`, both `current_link_width=8` at 32.0 GT/s, read 2026-10-18). No XGMI. Separate IOMMU groups — `03:00.0` → group **15**, `07:00.0` → group **20** (`/sys/bus/pci/devices/.../iommu_group`, read 2026-10-18) |
| P2P | **Live and in use.** The running engine prints "I device: the 2 cards reach each other's memory directly (peer to peer)" (`podman logs frank-archon`, 2026-10-15 boot). The 2026-09-20 census had predicted the opposite — ACS `ReqRedir+` on both root ports, so traffic hairpinning at the root complex — and that prediction is superseded; see [[Dual-R9700-Cooperation]] |
| P2P | **Live and in use.** The running engine prints "I device: the 2 cards reach each other's memory directly (peer to peer)" (`podman logs frank-archon`, 2026-10-15 boot). The 2026-09-20 census had predicted the opposite — ACS `ReqRedir+` on both root ports, so traffic hairpinning at the root complex — and that prediction is superseded; see |
| PCI | 03:00.0 and **07:00.0** (were `03:00.0` / `0b:00.0` on the B650-E) |
| iGPU | Raphael (gfx1036) at **11:00.0** — enumerates as ROCm device 2 (rocm-smi GPU[2], 512 MiB). The retired llama.cpp trios hid it with `HIP_VISIBLE_DEVICES=0,1`; the live radiance engine instead enumerates all three AQL devices and ignores it ("3 devices [0:gfx1201 1:gfx1201 2:gfx1036], using 2 of 3, ignoring gfx1036") |
@@ -104,8 +104,7 @@ RCCL-backed rebuild needs no new host packages.
| Serving stack | radiance 1.0.13 container (`stilldeadcode/radiance@sha256:4c3f484564…`), TP=2 across both R9700s — [[frank-archon]]. The llama.cpp `fb27a525` build at `~/GitPackages/llama.cpp-dflash2/build-rdna4-rocm714` is the retired frank-medium artifact |
The stale AMD-repo ROCm 6.3.3 packages (167, `~24.04`-suffixed) that used to
hijack `libhsa-runtime64` were purged on 2026-09-15 — see
[[frank-medium-rocmfpx]].
hijack `libhsa-runtime64` were purged on 2026-09-15 —
### Active Services
@@ -117,12 +116,12 @@ host-network podman container answering `curl 127.0.0.1:7770/health` →
The services this page once called active are gone: `frank-medium.service`
(llama-server, 32:33 layer split + DFlash2 drafter) is `not-found` — that rung
ran 2026-09-16 → 09-20 and is documented in [[frank-medium-llamacpp-dflash2]];
ran 2026-09-16 → 09-20 and is documented in
`frank-zealot.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU
swap. Its archived unit was moved to `~/.AGENTTRASH/20260915-frank-zealot-retire/host-units/`
and the vllm-radiance replica trio to `~/.AGENTTRASH/20260916-vllm-radiance-retire/`
— neither path exists today (checked 2026-10-18), so the archive location is
unverified. See [[deployment]].
unverified. See .
## Hardware Relevance
@@ -134,7 +133,7 @@ unverified. See [[deployment]].
tokens at 32:33, ~6.3/~5.2 GiB headroom) is history. Either way a batch or
context change needs a re-fit and not a guess.
- 246 GiB system RAM is **not** headroom: the live shape pins a 43.88 GiB host
pool per rank (`FRANK_ARCHON_HOST_POOL_MIB=44928`,
pool per rank (`FRANK_ARCHON_HOST_POOL_MIB=44928`
`scripts/frank-archon.env:184`) plus the 47.68 GiB n-gram/PLE table in RAM,
and `free -g` showed **90 GiB available** with the engine loaded (read
2026-10-18). The frank-medium claim that "nothing in the serving path is
@@ -147,6 +146,6 @@ unverified. See [[deployment]].
## See Also
- [[frank-archon]] — the active serving runbook
- [[Dual-R9700-Cooperation]] — why the PCIe path matters, and what the swap changed
- [[deployment]] — the deploy flow
- why the PCIe path matters, and what the swap changed
- the deploy flow
- [[Shared Memory Architecture Dankclaw]] — for contrast, Dankclaw's unified memory
serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md
-327
View File
@@ -1,327 +0,0 @@
---
type: research
title: Dual R9700 Cooperation
description: Why frank-medium's two Radeon AI PRO R9700s could not run tensor-parallel on the old B650-E board (PCIe atomic routing), what the 2026-09-20 ProArt B850-Creator swap unblocked, and what still has to be probed.
tags:
- frank-medium
- rocm
- rccl
- pcie-atomics
- tensor-parallelism
- dankserver
timestamp: 2026-09-16
---
# Dual R9700 Cooperation
Dankserver's two R9700s run **one tensor-parallel engine across both cards** —
today that is `frank-archon` under radiance 1.0.13, TP=2
([[frank-archon]]). When this page was written the endpoint
was `frank-medium`: first llama.cpp's layer split
([[frank-medium-llamacpp-dflash2]], 2026-09-16 → 09-20), then from 2026-09-20
one vLLM-radiance process at `tensor_parallel_size 2`
([[frank-medium-radiance-mxfp4]], 2026-09-20 → 09-26). This page records why
true tensor parallelism was impossible on the previous board (TUF GAMING B650-E
WIFI), and what the 2026-09-20 board swap changed.
## The short answer
**The wall is gone.** On the B650-E, card #2 sat behind the chipset, whose
bridges could not route PCIe atomic operations. RCCL's kernels carry a hidden
hostcall-buffer parameter and ROCr refuses to supply it to a device without
atomics, so *no* RCCL collective or P2P operation could be dispatched on that
card — see the [measured matrix](#measured-matrix) and the
[2026-09-16 table](#the-same-gate-as-this-board-reports-it-2026-09-16) below.
On 2026-09-20 the board became an **ASUS ProArt B850-Creator WiFi NEO**, which
bifurcates the CPU's GPP ports **x8/x8**: the board-swap path this page
recommended, at B850 rather than X870E money. Both cards now hang off CPU root
ports (`00:01.1`, `00:01.3`), every bridge on both chains reads
`AtomicOpsCap: Routing+`, and **both endpoints read `AtomicOpsCtl: ReqEn+`** —
the exact precondition this page's own test plan named, and it is now
[measured](#measured-verdict-2026-09-20) rather than assumed.
What that bought:
- **Both cooperation modes now work, and they were both run.** vLLM TP=2 booted
and served (its R4D allreduce needs no RCCL collective at all), and llama.cpp
`--split-mode tensor` booted and served against an RCCL-backed build — so a
collective HAS been dispatched on this host since the swap.
- **TP=2 is the production shape.** It wins on every axis measured, by 2.1x on
decode and 3.6x at 4-way concurrency, and it replaced the layer split on the day
of measurement. The payload has moved since: the MXFP4 `frank-medium` shape is
retired 2026-09-26, and what runs today is the iq4r publisher under radiance
1.0.13 ([[frank-archon]]).
- **The drafter, not the collective, is what llama.cpp lost.** Its tensor mode
could not carry the DFlash2 drafter, which is the honest caveat on the
comparison below.
- **The bandwidth story moved too.** Both cards ride Gen5 x8 (~31.5 GB/s each):
card 0 effectively unchanged (was Gen4 x16), card 1 up ~16x (was Gen4 x1,
~1.97 GB/s). The M.2→PCIe riser plan is dead — `PCIEX16_2` holds card #2 now
and M.2_2 is free.
## The gate, in source
```cpp
void Device::checkAtomicSupport() { // rocclr/device/rocm/rocdevice.cpp:181
std::vector<amd::Device::LinkAttrType> link_attrs;
link_attrs.push_back(std::make_pair(LinkAttribute::kLinkAtomicSupport, 0));
if (findLinkInfo(cpu_agent_info_->fine_grain_pool, &link_attrs)) {
if (link_attrs[0].second == 1) {
info_.pcie_atomics_ = true; // no else branch
}
}
}
```
The flag is a **queried hardware property**, consumed at
`rocvirtual.cpp:3954` (the hostcall/AQL dispatch path), in `rocblit.cpp` (SDMA
copy paths) and exported as `hipDeviceProp.hostNativeAtomicSupported`. There is
no environment variable, module parameter, or sysfs toggle around it — verified
by grepping ROCclr for `getenv`/`HSA_*`/`ROCR_*`/`AMD_*` in the gate's
neighborhood: none. Writing the endpoint's Device Control 2 bit 6 by hand
(`setpci -s 0b:00.0 CAP_EXP+28.w=0440`) does nothing because the value is
consumed at driver probe time.
What has to be true upstream of it: every PCIe bridge on the GPU's path to the
CPU must advertise `AtomicOpsCap: Routing+`. On this board card #2's path
includes `05:00.0` and `06:09.0`, both `Routing-` — a **read-only capability
bit**, so no software can enable it.
### The same gate, as this board reports it (2026-09-16)
`lspci -vv` on dankserver. Card 0's chain is root port `00:01.1` → `01:00.0` →
`02:00.0` → `03:00.0`; card 1's is `00:02.1` → `05:00.0` → `06:00.0` →
`06:09.0` → `09:00.0` → `0a:00.0` → `0b:00.0`:
| Device | Role | `AtomicOpsCap` | `AtomicOpsCtl` | Link |
|---|---|---|---|---|
| `00:01.1` | CPU root port → PCIEX16_1 | Routing+ | — | x16 |
| `03:00.0` | **card 0** | 32bit+ 64bit+ | **ReqEn+** | x16 |
| `00:02.1` | CPU root port → chipset | **Routing+** | — | x4 |
| `05:00.0`, `06:00.0`, `06:09.0` | B650 bridges | **Routing−** | — | x1 at `06:09.0` |
| `09:00.0`, `0a:00.0` | PCIe switch ports | Routing+ | — | x1 (downgraded) |
| `0b:00.0` | **card 1** | 32bit+ 64bit+ | **ReqEn−** | x16 |
Three things fall out of that table, none of them in the source alone:
- **The CPU side is fine** — `00:02.1` is `Routing+`. The wall is inside the
chipset, not the platform, which is why a slot on CPU lanes is the fix.
- **The endpoint's own bit differs between the cards, and that is the kernel's
doing.** `pci_enable_atomic_ops_to_root()` walks the bridge chain and refuses
to set Requester Enable when any bridge cannot route atomics. ROCclr's
`pcie_atomics_` and the endpoint's `ReqEn` are two faces of one cause, and it
makes `lspci -vv -s <bdf> | grep AtomicOpsCtl` a one-command diagnosis: on
this host card 0 reads `ReqEn+` and card 1 `ReqEn−`.
- **Card 1's effective link is x1**, not the x16 its own capability block
advertises: `06:09.0` trained at x1, so ~2 GB/s is the real ceiling. That is a
bandwidth fact rather than an atomics one, and it only matters for
collectives — a 27B TP=2 allreduce is ≈5.6 ms/token at x1 against ≈1.4 ms at
x4. The layer split's boundary copy does not care.
## Re-measured 2026-09-20 (ProArt B850-Creator WiFi NEO)
`lspci -vv` after the board swap. Card 0's chain is `00:01.1` → `01:00.0` →
`02:00.0` → `03:00.0`; card 1's is `00:01.3` → `05:00.0` → `06:00.0` →
`07:00.0`. **No chipset bridge is in either path** — `00:02.1` now carries only
LAN/SATA/WiFi:
| Device | Role | `AtomicOpsCap` | `AtomicOpsCtl` | Link |
|---|---|---|---|---|
| `00:01.1` | CPU GPP → PCIEX16_1 | Routing+ | — | Gen5 x8 |
| `01:00.0` | card 0 switch upstream | Routing+ | — | Gen5 x8 |
| `02:00.0` | card 0 switch downstream | Routing+ | — | Gen5 x16 |
| `03:00.0` | **card 0** | 32bit+ 64bit+ | **ReqEn+** | Gen5 x16 (on-card) |
| `00:01.3` | CPU GPP → PCIEX16_2 | Routing+ | — | Gen5 x8 |
| `05:00.0` | card 1 switch upstream | Routing+ | — | Gen5 x8 |
| `06:00.0` | card 1 switch downstream | Routing+ | — | Gen5 x16 |
| `07:00.0` | **card 1** | 32bit+ 64bit+ | **ReqEn+** | Gen5 x16 (on-card) |
- Both endpoints read `CAP_EXP+28.w = 0x1440` — bit 6 set, which is the
kernel's `pci_enable_atomic_ops_to_root()` having walked a chain with no
`Routing-` bridge. Same one-command diagnosis, opposite answer.
- `dmesg` agrees on bandwidth: `252.056 Gb/s available PCIe bandwidth, limited
by 32.0 GT/s PCIe x8 link at 0000:00:01.1` (and `.13`) — ~31.5 GB/s per
card. DMI confirms the bifurcation: `PCIEX16(G5)_2`, "8x or x8".
- `rocm-smi --showtopo` reported `PCIE` between the cards (Navi 48 has no XGMI),
they were in separate IOMMU groups (15 and 20 — re-read 2026-10-18: still 15
and 20), and both root ports had ACS `ReqRedir+`, so this page inferred P2P
would hairpin at the root complex. **That inference is superseded:** the live
engine reports the two cards reaching each other's memory directly, which
[[frank-archon]] and [[Dankserver System Specs]] record. Plain P2P traffic
measured 45.27 GB/s bidirectional on the *old* board — DMA was never the gate,
atomics were.
- Kernel 7.0.0-31-generic with `iommu=pt` then; this host runs 7.0.0-38-generic
with `iommu=pt` today (`/proc/cmdline`, read 2026-10-18). Both were ruled out
as causes ([above](#ruled-out-empirically-do-not-re-derive)) and this host's
IOMMU posture is unchanged.
**Still unproven (as of 2026-09-20).** The gate that matters is ROCclr's
`checkAtomicSupport()` reading the `kLinkAtomicSupport` link attribute; the ReqEn
bit is the same cause seen from the kernel side, but no RCCL kernel had been
launched since the swap. Until the 2-rank probe runs — or a `--split-mode row`
boot that does *not* log `internal AllReduce init failed ... falling back to
meta-backend butterfly` — "collectives work here" was a strong expectation, not a
measurement. **RCCL never got that measurement**; what settled the operational
question instead is production traffic — radiance's TP=2 wire carries cross-rank
all-reduces over the direct P2P path and prints its own precision, and the
measured `wht6` banner with its exact-step bound is on [[frank-archon]].
## Measured matrix
Bare 2-rank `torch.distributed` (`backend=nccl`) on the two R9700s, no vLLM, no
model. Rank 0 = card #1 (CPU lanes), rank 1 = card #2 (chipset):
| Operation | rank 0 (card #1) | rank 1 (card #2) |
|---|---|---|
| `broadcast` | OK | **FAIL** `DistBackendError` |
| `all_reduce` | OK | **FAIL** |
| `barrier` | OK | **FAIL** |
| `send` / `recv` (P2P) | blocked, then timeout | **FAIL** |
| `all_gather` (after failure) | group unusable | — |
The `AMD_LOG_LEVEL=3` trace names the cause: `Pcie atomics not enabled, hostcall
not supported` → `AQL dispatch failed!` → `hipLaunchKernel` returns
`hipErrorIllegalState` for `ncclDevKernel_Generic_4(...)`. Nothing in vLLM or
RCCL mentions atomics; the runtime log is the only place it appears.
**`send`/`recv` failing is the decisive result**: it kills pipeline parallelism
as an option, because vLLM's PP stage handoff is exactly that primitive.
## Software escapes: closed, and the last one is now moot
Every row below was written under the old board. The last row — the only door
that was ever open — stopped mattering on 2026-09-20, because the hardware gate
it existed to route around no longer applies to this host.
| Idea | Finding |
|---|---|
| vLLM's IPC custom allreduce | **Gated off by a table keyed to CUDA sm versions** — `{9.0, 10.0, 10.3, 10.7}`; ROCm reports capability `12.0` for gfx1201, so the gate is `False`. `custom_ar` itself is compiled into the image. |
| ...and if that table were patched | Still insufficient: `broadcast`, `barrier`, and the TP logits `all_gather` have no non-RCCL path. |
| The radiance fork's own allreduce | Already present and *on by default* (`RADIANCE_USE_R4D_AR`, size gate `RADIANCE_AR_MAX_KB`): a one-shot P2P push allreduce wrapping `CudaCommunicator.all_reduce`, exchanging IPC handles over the **gloo** group. With the gate raised it covers prefill and decode, and `ncclDevKernel` disappears from the profile. But it does not intercept `PyNcclCommunicator.__init__`'s **warmup allreduce**, nor `pynccl_comm.all_gather` — both direct RCCL calls. |
| NCCL symmetric memory | Unavailable: the shipped RCCL advertises NCCL 2.30.4 but exposes no `ncclDevComm*` API, so torch's symmetric-memory path cannot run. |
| Prefill/decode disaggregation | Needs a KV-transfer connector plus coordination collectives; both out. |
| gloo for GPU work | CPU/TCP collectives — orders of magnitude too slow for per-layer TP. |
| **Rebuild RCCL so its kernels do not declare a hostcall buffer** | **OPEN — unverified.** The claim is that the hostcall requirement is triggered by device-side `printf`/`assert`, and that `-mprintf-kind=buffered` avoids it. Counter-evidence: RCCL's device source at `src/device/` contains **no live** `printf`, `assert`, or `__assertfail` (checked), so what makes its kernels declare `HiddenHostcallBuffer` is unexplained. The claim traces to search-result summaries, not a primary source. Cheap to settle once a host is available: recompile librccl with `-mprintf-kind=buffered`, dump a kernel's declared arguments (`roc-obj-extract` + `llvm-readelf --notes`) and look for `hidden_hostcall_buffer`; if it is gone, a 2-rank probe should dispatch. **Stop and mask the service first** — see the note below. |
## Bandwidth: why x4 is enough
A 27B TP=2 allreduce at batch 4 moves roughly
$2 \times \text{batch} \times \text{hidden} \times 2\ \text{bytes} \approx 160\ \text{KB}$
per layer per forward pass; across ~62 layers that is **≈10 MB per token-step**,
or **~1.4 ms at PCIe 4.0 x4** (~7.9 GB/s), and under 0.4 ms at the Gen5 x8
~31.5 GB/s each card now has. TP is compute-bound here, not interconnect-bound.
That is why the cheap option (the M.2 riser) and the expensive one (the board
swap) were both considered adequate on bandwidth alone — and why the board that
landed, at x8/x8, clears the arithmetic with room to spare.
## Ruled out empirically (do not re-derive)
- Kernel versions 7.0.0-28 / -30 / -31 — byte-identical failure.
- `iommu=pt`, and `amd_iommu=off` is unrelated (that was a Dankclaw prefill knob).
- P2P hardware: 45.274 GB/s bidirectional between the cards works fine — plain
memory traffic is not the problem; *atomic* routing is.
- A from-source librccl rebuild: identical failure at the identical call path.
- Every NCCL transport/algorithm knob.
- A one-card test proves nothing: NCCL short-circuits at `world_size=1`, which
is exactly why the two replicas work at all.
- Independent corroboration: StillDeadcode/vllm-radiance Codeberg issue #22 —
different host, same image, kernel 7.0.0-30, same `enqueue.cc` failure.
## Measured verdict (2026-09-20)
Both cooperation modes were run on this host on 2026-09-20 — the same salted
10-prompt battery (`scripts/bench-conversation.py`), the service stopped and
masked between arms. The arm table is [[frank-medium-radiance-mxfp4]]'s bench of
record (arms R = radiance TP=2 + DFlash2 as deployed, I = the llama.cpp layer
split that preceded it, L = llama.cpp `--split-mode tensor` against RCCL); this
page keeps only the verdict. **TP=2 won on every axis — roughly 2.1× decode and
3.6× at 4-way concurrency over the layer split — and replaced the layer split the
day of measurement.**
Arm L carries a caveat no flag can remove: llama.cpp's tensor mode could not load
the DFlash2 drafter (it aborts at `ggml-backend.cpp:941` — a draft weight
pre-allocated on the non-primary device has no backend once `n_copies == 1`), so
it decoded **without** speculation while the other arms decoded with it; and part
of R's margin over I is the 4-bit MXFP4 checkpoint, not the parallelism.
Two numbers elsewhere on this page are therefore not interchangeable:
- The 272 t/s recorded 2026-09-16 is **prefill**, on a **butterfly** allreduce,
with RCCL off, on the old board. It was never a TP decode number, and its
paired decode figure is in "The third path" below.
- The butterfly tensor arm and the RCCL tensor arm a day later are the same shape
family on different collective paths; the measured pair lives with
[[frank-medium-radiance-mxfp4]]'s arms.
RCCL needs PCIe atomics on *every* participating GPU — the gate the old board
failed. vLLM's R4D one-shot allreduce dispatches no collective at all, which is
why TP=2 runs today rather than merely "being unblocked": the image ships TheRock
RCCL, and TheRock carries no gfx1201 device code (`invalid device function`), so
radiance could not have used RCCL on this host even with atomics routed. Ubuntu's
RCCL 2.27 does carry gfx1201 kernels, and that is what arm L linked against.
**Rule for any future multi-GPU collective experiment: stop and mask the
production service first**, per the benchmarking policy, or run it on a host that
is not serving users. The production service is `frank-archon`, a **root** unit,
so on Dankserver that is `sudo systemctl stop frank-archon && sudo systemctl
mask frank-archon` (`scripts/frank-archon.service`; never `disable` — that
deletes the unit file, and `mask` is what the bench policy prescribes). The
2026-09-20 wording below — `systemctl --user stop frank-medium && systemctl
--user mask frank-medium`, frank-medium being a rootless quadlet — is retired
with that service, which today reports `not-found` (read 2026-10-18).
This is not theoretical: on 2026-09-16 a bare RCCL P2P probe was run against the live
service; the host went hard down ~20 minutes later (no ARP, no ICMP, required a
power cycle). The mechanism is not proven, but a hung AQL dispatch wedging the
GPU driver is a plausible cause and this host has precedent for going
unreachable until a power cycle.
## The third path (production in the layer-split era): llama.cpp's layer split
Everything above is about *collectives*, and vLLM cannot proceed without them
in any of its parallelisms — TP needs a per-layer allreduce, and PP's stage
handoff is a p2p `send`/`recv` of the hidden state, the primitive this host
also fails. **llama.cpp's `--split-mode layer` needs no collective at all**:
layers are distributed across the cards and the boundary activation is a
plain per-backend copy, once per forward pass. No RCCL kernel is ever launched
on the second card, so the atomics gate never comes up.
That is what `frank-medium` ran from 2026-09-16 to 09-20
([[frank-medium-llamacpp-dflash2]]): one process, one model, both cards,
262144 context. Its measured layer-split rates are that page's; this host's
tensor arm under the butterfly fallback measured 272 t/s prefill / 26.3 tok/s
decode. **That tensor number is not a prediction for an RCCL-backed build** — it
is what tensor split costs when the allreduce degrades to a meta-backend
butterfly, and it is the number the board-swap path exists to retire.
The ranking therefore changed twice. On 2026-09-16: software already gets both
cards onto one model, so the hardware fix stopped being a capability question
and became a performance one — *tensor* parallelism, ~40 % decode, plus one
pooled KV instead of two independent 122k-token pools. On 2026-09-20 the
hardware half of that question was answered by the board swap; the llama.cpp
measurement half never ran, and `GGML_HIP_RCCL` still defaults `OFF` in
`scripts/build-llama-rdna4.sh:41` (re-read 2026-10-18) for exactly the reason
this page documents — moot while no llama.cpp shape is deployed on this host.
## What would change the answer
- ~~An RCCL build whose kernels stop declaring a hostcall buffer~~ — **moot
since 2026-09-20.** That door only mattered while the hardware gate could not
be passed; the board swap passes it. The `-mprintf-kind=buffered` hypothesis
and its counter-evidence stay in the escape table for the record.
- The symmetric-kernel path inside RCCL, if a release ever ships the device API
(`ncclDevComm*`) that torch's symmetric-memory path needs. The shipped TheRock
nightly (`~/.local/share/rocm-7.14.0a20260624/lib/librccl.so.1`) still does
not.
- ~~Firmware or an AGESA update that makes the chipset bridges advertise atomic
routing~~ — arrived as new silicon instead.
- A GPU-side workaround: none found in ROCclr, and the gate is per-device,
queried once at probe.
## See Also
- [[Pipeline Parallelism/index|Pipeline Parallelism]] — two-machine research (the PP variant of this problem)
- [[Dankserver System Specs]] — hardware inventory
- [[Observability]] — the scrape/dashboard/alert stack around these endpoints
serverAppConfigs-docs/LLM/FastFlowLM NPU Inference.md
-448
View File
@@ -1,448 +0,0 @@
---
type: research
title: FastFlowLM NPU Inference
description: FastFlowLM (FLM), the AMD NPU-first LLM runtime for Ryzen AI XDNA2 chips. Fit analysis for the 50 TOPS NPU on Dankclaw, published benchmarks, verified install state, and a test runbook.
tags:
- fastflowlm
- npu
- xdna2
- dankclaw
- research
timestamp: 2026-09-03
---
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
> See [[frank-redo-2026-08-17]] and [[deployment]] for the active configuration.
> The `frank-zealot` name and port 52625 then belonged to the vLLM
> service on Dankclaw (nameistoken Quark W8A8); this page is archive.
>
> **The NPU endpoint is retired — 2026-09-11.** On Dankclaw today
> `frank-npu.service` is `masked`/`inactive` and nothing listens on 52625
> (`ss -ltn`, read 2026-10-18). The runtime is still installed
> (`fastflowlm 1.0.1`, `libxrt2`/`libxrt-npu2`, `/opt/fastflowlm`,
> `~/.config/flm`, `/usr/bin/flm` — all read 2026-10-18) and the device is
> usable again since the 2026-10-18 `iommu=pt` boot (`amdxdna` bound,
> `/dev/accel/accel0` present — [[dankclaw-iommu-translation-off]]), but no
> service runs on it. `frank-zealot` today is the llama.cpp PrismML endpoint on
> the RTX 3090 ([[frank-zealot]]). See [[frank-npu]]
> for the retirement record. Everything below is the 2026-08 research record.
# FastFlowLM NPU Inference
FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM was installed on Dankclaw, the NPU validated, and `frank-zealot` (port 52625) served in production — that arrangement ended with the 2026-09-11 retirement (see the banner above). See Promotion.
## Verdict (2026-08-16)
- The NPU is a complement, not a replacement. It fits dense models up to about 9B.
- Qwen3.8-27B (dense) cannot run on the NPU. No FLM build exists and the memory budget is too small.
- Qwen3.6-35B-A3B has an official FLM build (Q4_KS). AMD benchmarks it on a 96 GB machine. It probably does not fit the 30 GiB budget of Dankclaw.
- Realistic first candidates: Qwen3.5-9B, Qwen3-8B, gpt-oss-20b (at the budget edge).
- **Landed 2026-08-16:** gpt-oss-20b serves on the NPU at 19.3 t/s decode, concurrently with the PP head. See Measured Results.
- The memlock limit was the setup blocker. Fixed 2026-08-16 via `/etc/security/limits.d/99-flm-memlock.conf`.
## What FastFlowLM Is
FLM is an NPU-first inference runtime. It started as a 2025 university project (University of Rhode Island, Ken Qing Yang). The team joined AMD on 2026-07-17. The repository moved to the ROCm organization on 2026-08-11 as v1.0.0.
Key properties, per the project README and docs:
- Runs fully on the NPU. No GPU or CPU load during inference.
- CLI mode (`flm run <tag>`) and server mode (`flm serve <tag>`).
- Server mode exposes a REST API and an OpenAI-compatible API. Default port is 52625.
- Models are not GGUF. FLM downloads pre-compiled kernel packs (built with AMD IRON + AIE-MLIR) from Hugging Face.
- Context lengths up to 256k tokens on listed models.
- Vision, audio, embedding, and MoE model support.
- Runtime is about 17 MB. Orchestration code is MIT-licensed. Kernels are free for commercial use.
- Models store in `~/.config/flm/` (Linux). Override with `FLM_MODEL_PATH`.
## NPU Hardware on Dankclaw
Dankclaw runs an AMD Ryzen AI MAX+ 395 (Strix Halo). The chip integrates an XDNA2 NPU
with 50 TOPS of compute and, crucially, **no dedicated VRAM** - it shares system memory
via the IOMMU SVA mapping. The budget this page reasons about - 29 GiB usable at
4800 MT/s LPDDR5X - is the 2026-08 host state; the live figure is [[Dankclaw System Specs]].
Live verification on Dankclaw, 2026-08-16:
```
$ flm validate
[FLM] Fetching models from: /opt/fastflowlm/share/flm/model_list.json
[Linux] Kernel: 7.0.0-29-generic
[Linux] NPU: /dev/accel/accel0 with 8 columns
[Linux] NPU FW Version: 1.1.2.65
[Linux] amdxdna version: 0.7
[ERROR] Memlock limit is too low (8MB). Please raise the limit or set to infinity.
```
- Kernel 7.0.0-29-generic shipped the `amdxdna` driver in-tree, loaded. The host now runs 7.0.0-38-generic and the driver is still bound (`lspci -k -s c6:00.1` → `Kernel driver in use: amdxdna`, read 2026-10-18).
- Firmware 1.1.2.65 meets the 1.1.0.0 minimum.
- The XRT runtime packages (`libxrt2`, `libxrt-npu2`) are installed — still installed on 2026-10-18 (`dpkg -l` → `libxrt2 1:2.21.75+dfsg-4`, `libxrt-npu2 1:2.21.75+dfsg-4`).
- The only failure is the memlock ulimit (8 MB). FLM needs `unlimited`.
- Ubuntu 26.04 is not on the FLM supported-distro list (24.04, 25.10, Arch, generic). Validation passes anyway. Treat 26.04 as "generic Linux, working".
## Memory Model
The NPU allocates from system RAM. AMD/Microsoft enforce an internal cap of about half of total system RAM for NPU use. The gpt-oss model card states a cap of about 15.6 GB on a 32 GB machine, which is half.
Budget math for this fleet:
| Machine | RAM | NPU budget (approx. half) |
|---------|-----|---------------------------|
| Dankclaw (Strix Halo) | 30 GiB | ~15 GiB |
| Dankserver (Ryzen 7 7700X) | 62 GB | no NPU |
| FLM bench box (Kraken Point) | 32 or 96 GB | ~16 or ~48 GB |
The half-of-RAM gate scales with host RAM - live figures in [[Dankclaw System Specs]]
and [[Dankserver System Specs]]. The live gate on Dankclaw is still `flm validate`
(it owns the current number).
Weight sizes (approximate, Q4-class):
| Model | Quant | Weights | Fits ~15 GiB budget? |
|-------|-------|---------|----------------------|
| Qwen3.5-9B | Q4_1 | ~5.5 GB | yes |
| Qwen3-8B | Q4_1 | ~5 GB | yes |
| gpt-oss-20b (MoE, 3.6B active) | Q4_1 | ~15.1 GB | at the edge |
| Qwen3.6-35B-A3B (MoE, 3B active) | Q4_KS | ~20 GB | no, unproven at 30 GiB |
| Qwen3.8-27B (dense) | any Q4 | ~16 GB+ | no build exists |
Do not load a model larger than the budget first. An oversized load can starve the unified pool and crash the box while production services still run.
## Supported Models (Qwen family, plus gpt-oss)
All Qwen cards use Q4_1 unless noted. Context lengths are the FLM-published maxima.
| Tag | Base model | Context max | Notes |
|-----|-----------|-------------|-------|
| `qwen3:0.6b` | Qwen3-0.6B | 32k | thinking toggle |
| `qwen3:1.7b` | Qwen3-1.7B | 32k | thinking toggle |
| `qwen3:4b` | Qwen3-4B | 32k | tool calling |
| `qwen3:8b` | Qwen3-8B | 32k | tool calling |
| `qwen3-tk:4b` | Qwen3-4B-Thinking-2507 | 256k | thinking |
| `qwen3-it:4b` | Qwen3-4B-Instruct-2507 | 256k | tool calling |
| `qwen3vl-it:4b` | Qwen3-VL-4B-Instruct | 256k | vision |
| `qwen2.5-it:3b` | Qwen2.5-3B-Instruct | 32k | |
| `qwen2.5vl-it:3b` | Qwen2.5-VL-3B-Instruct | 256k | vision |
| `qwen3.5:0.8b` | Qwen3.5-0.8B | 256k | vision, thinking toggle |
| `qwen3.5:2b` | Qwen3.5-2B | 256k | vision, tool calling |
| `qwen3.5:4b` | Qwen3.5-4B | 256k | vision, tool calling |
| `qwen3.5:9b` | Qwen3.5-9B | 256k | vision, tool calling |
| `qwen3.6-moe:35b-a3b` | Qwen3.6-35B-A3B | 256k | **Q4_KS**, MoE, vision, thinking, tool calling |
| `gpt-oss:20b` | gpt-oss-20b | 128k | Q4_1, ~15.1 GB in FLM |
Other families: LLaMA, DeepSeek, Gemma, MedGemma, TranslateGemma, LiquidAI/LFM, Phi, Nanbeige, Whisper, EmbeddingGemma, SmolVLA.
## Published Benchmarks (FLM, vendor numbers)
All vendor results come from AMD Ryzen AI 7 350 (Kraken Point, 40 TOPS NPU) boxes under the default NPU power mode (Performance). Our NPU is a 50 TOPS Strix Halo XDNA2. Treat these numbers as directional, not as our expected speed.
Qwen3.5, FLM v0.9.38, Kraken Point 32 GB DRAM. Decode t/s by starting context length:
| Model | 1k | 2k | 4k | 8k | 16k | 32k |
|-------|-----:|-----:|-----:|-----:|-----:|-----:|
| Qwen3.5-0.8B | 39.2 | 38.0 | 36.3 | 33.1 | 28.1 | 21.6 |
| Qwen3.5-2B | 26.8 | 26.2 | 25.4 | 23.7 | 21.3 | 17.0 |
| Qwen3.5-4B | 15.0 | 14.6 | 14.2 | 13.3 | 11.8 | 9.6 |
| Qwen3.5-9B | 9.3 | 9.2 | 9.0 | 8.5 | 7.8 | 6.9 |
Prefill t/s for the same run:
| Model | 1k | 2k | 4k | 8k | 16k | 32k |
|-------|-----:|-----:|-----:|-----:|-----:|-----:|
| Qwen3.5-0.8B | 983 | 1257 | 1471 | 1584 | 1579 | 1447 |
| Qwen3.5-2B | 803 | 1004 | 1142 | 1223 | 1225 | 1151 |
| Qwen3.5-4B | 378 | 440 | 479 | 493 | 487 | 450 |
| Qwen3.5-9B | 284 | 333 | 362 | 379 | 378 | 357 |
Qwen3.6-35B-A3B (Q4_KS), FLM v0.9.45, Kraken Point **96 GB DRAM**:
| Metric | 1k | 2k | 4k | 8k | 16k | 32k |
|--------|-----:|-----:|-----:|-----:|-----:|-----:|
| Decode t/s | 13.65 | 13.41 | 13.09 | 12.51 | 11.24 | 9.51 |
| Prefill t/s | 78.98 | 118.04 | 156.43 | 197.93 | 218.84 | 221.96 |
- PP and NPU together cost 2.2 GiB + NPU model + ~2 GiB of headroom. The then-current
PP split (head + 3090 worker, 2026-08 era - see [[frank-redo-2026-08-17]]) decoded at
14-18 t/s for the 27B dense model, so an NPU offload would free the head or upgrade
the worker's model.
The vendor also claims "over 10x more power-efficient" versus GPU. That is a marketing claim. We have not measured it.
## Fit to the Frank Program
| Our model | NPU verdict |
|-----------|-------------|
| Qwen3.8-27B UD-Q8_K_XL (Dankclaw, dense) | No. No FLM build. ~16 GB+ even at Q4. Exceeds the ~15 GiB budget. |
| Qwen3.6-35B-A3B (Dankserver, MoE) | Unproven. Official build exists. Vendor bench ran on 96 GB RAM. A 30 GiB box has ~15 GiB for the NPU. Test last, with the service stopped. |
| Qwen3.5-9B / Qwen3-8B | Yes, the realistic tier. ~5-6 GB. Leaves room for KV and the OS. |
| gpt-oss-20b (MoE) | **Landed.** Weights are 13.8 GB on disk (q4nx). The model holds ~15 GiB resident on the NPU. Decode is 19.3 t/s. See Measured Results. |
Proposed role: a small, always-on, low-power endpoint for triage, summarization, embeddings, or agent side-tasks on the Dankclaw NPU. It frees the iGPU and the 3090 for the 27B/35B workloads. It does not replace either.
## Install State on Dankclaw (verified 2026-08-16)
- `fastflowlm 1.0.1` installed (deb, upgraded from 0.9.45-12-ga6bd056 on 2026-08-17). `flm` at `/usr/bin/flm`. The 0.9.45 runtime refused the qwythos pack, which declares `flm_version` 1.0.0.
- `libxrt2` / `libxrt-npu2` (Xilinx Runtime) installed.
- `/dev/accel/accel0` present. `flm validate` passed fully. (Re-read 2026-10-18: `/dev/accel/accel0` is still present and `fastflowlm 1.0.1` is still installed, but the endpoint is retired — `frank-npu.service` masked, no listener on 52625.)
- Memlock fixed: `/etc/security/limits.d/99-flm-memlock.conf` sets soft and hard memlock to unlimited. A fresh sshd session picks it up via pam_limits. No reboot needed.
- `gpt-oss:20b` pulled (4/4 files verified, 13.8 GB `model.q4nx` + tokenizer) into `~/.config/flm/models/GPT-OSS-20B-NPU2/`.
- `qwythos:9b` (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2) installed 2026-08-17 via `flm-add.py` (downloaded from the HF repo; stdlib-only, no pip needed). Tag registered in the user registry `~/.config/flm/model_list.json` (a copy of the v1.0.1 system registry + the custom entry); kernels symlinked `~/.config/flm/xclbins/Huihui-...-NPU2 -> /opt/fastflowlm/share/flm/xclbins/Qwen3.5-9B-NPU2`. The service env exports `FLM_CONFIG_PATH` and `FLM_XCLBIN_PATH` so the tag resolves (without them flm hits the unknown-tag fallback, see the incident below).
- `flm serve gpt-oss:20b --host 0.0.0.0 -p 52625 --pmode performance` runs as a nohup process. It is not a systemd unit. It does not survive a reboot.
- The `llama-server` (frank-dragoon, the PP head) holds an fd on `/dev/accel/accel0` from its own startup (device enumeration, not compute). FLM opened the same device concurrently without conflict. Verified live 2026-08-16.
## Measured Results (2026-08-16, gpt-oss-20b on the Dankclaw NPU)
Conditions: `--pmode performance`, PP head (27B, port 7770) serving concurrently, 5.5 GiB system RAM headroom steady-state, NPU resident footprint ~15 GiB.
| Run | Prompt tok | Compl tok | Wall s | Prefill t/s | Decode t/s | KV occ % |
|-----|-----------:|----------:|-------:|------------:|-----------:|---------:|
| 1 (smoke) | 74 | 63 | 7.2 | 19.5 | 19.3 | 1.7 |
| 2 (160-tok prompt) | 161 | 256 | 17.4 | 40.8 | 19.3 | 5.1 |
| 3 (95-tok prompt, 512 out) | 95 | 512 | 30.8 | 24.3 | 19.3 | n/a |
Decode is flat at 19.3 t/s across all three runs. Output includes `reasoning_content` (gpt-oss is a reasoning model). At `max_tokens 256` one run spent its whole budget thinking and returned empty content. Give it 500+ tokens or strip reasoning in the client.
Long-context prefill is not yet measured (runs 1-3 used under 200 prompt tokens). The 35B-A3B step remains untested.
## Model Bakeoff (measured 2026-08-16, 4 models)
All runs on the Dankclaw NPU (50 TOPS Strix Halo), single stream, `--pmode performance`, PP head (frank-dragoon) serving concurrently on 7770. Decode t/s is the FLM-reported rate. Vision accuracy = ground-truth elements out of 4 (text SPECTRUM, red circle left, blue square right, three green triangles bottom).
| Model | Tag | Weights | Decode t/s (short) | 8k prefill t/s | 8k decode t/s | Vision decode t/s | Vision accuracy | Max ctx (registry) |
|-------|-----|---------|-------------------:|---------------:|--------------:|------------------:|----------------|--------------------|
| Gemma4-E4B | `gemma4-it:e4b` | 6.8 GB + 912 MB audio | 12.2 | 582 | 9.9 | 11.7 | 4/4 | 32768 |
| Qwen3.5-9B | `qwen3.5:9b` | 7.3 GB | 7.2 | 343 | 6.6 | 7.3 | 4/4 | 32768 |
| GPT-OSS-20B | `gpt-oss:20b` | 13.8 GB | 19.3 | 378 | 16.9 | n/a (no vision) | 0/4 | **8192** |
| Qwen3.6-35B-A3B | `qwen3.6-moe:35b-a3b` | 22.2 GB | 12.7 | 207 | 11.2 | 12.7 | 4/4 | 32768 |
Notes:
- gpt-oss:20b is the fastest decoder (MoE, 3.6B active) but its KV cache fills at ~8k tokens (98.85% occupancy at 7,926; requests past ~8k return 400). It has no vision encoder.
- 35B-A3B loads 28-29 GiB resident. It works but leaves ~1.5 GiB headroom with the PP head running. Do not run it alongside the head at full load.
- Vision prefill for the two vision models with ~720 token image input: gemma 141 t/s, qwen3.5 176 t/s, 35B-A3B 68 t/s.
- All four comply with the same image at 4/4 elements except gpt-oss, which cannot see it.
## Promotion (2026-08-16)
`frank-zealot` then ran the NPU endpoint on Dankclaw (2026-08; the name was reused
twice after this window - see the banner - and the NPU unit today is `frank-npu`,
masked). Unit: `scripts/frank-zealot.service` (symlinked into `/etc/systemd/system`),
wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-zealot.env`.
`LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap
`FLM_MODEL` in the env to switch.
### Boot policy (2026-08-17)
The `frank-zealot` unit is disabled for boot. Only the PP pair (`frank-dragoon` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw:
```
sudo ln -sf /home/dank/vllmScripts/scripts/frank-zealot.service /etc/systemd/system/frank-zealot.service
sudo systemctl daemon-reload
sudo systemctl enable frank-zealot
```
Live config (2026-08-16): `FLM_PMODE=turbo`, `FLM_ASR=1` (loads Whisper-V3-Turbo-NPU2; exposes `/v1/audio/transcriptions`), `FLM_CTX_LEN=131072` (verified max — Gemma4 natively supports 128K and the NPU budget fits it; KV occupancy measured at 0.11% for a 123-token exchange, decode 12.0 t/s, 13 GiB headroom).
The PP split was renamed `frank-pp-head` → `frank-dragoon` (2026-08-16, commit `48142f5`).
Model swap (2026-08-16, commit `50c55ee`): `FLM_MODEL=qwen3.6-moe:35b-a3b`, `FLM_CTX_LEN=262144` (native max), `FLM_PMODE=turbo`, `FLM_ASR=1` kept. Decode measured 12.5-13.1 t/s. The Gemma4 live config above (131072 ctx) applied to `gemma4-it:e4b`. Swap `FLM_MODEL` and `FLM_CTX_LEN` back to restore it.
Model swap (2026-08-17, commit `c85ff6c`): `FLM_MODEL=qwythos:9b` (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2, Q4NX 7.76 GB). The pack installs through `flm-add` (user registry `~/.config/flm/model_list.json` + user xclbins `~/.config/flm/xclbins`, kernels symlinked from official `Qwen3.5-9B-NPU2`). Runtime upgraded to FLM 1.0.1 first (see Install State). `FLM_CTX_LEN=262144` native max (Qwen3.5-9B hybrid: 8 of 32 layers full attention, so KV stays small; occupancy 0.008% at 21 tokens). `FLM_PMODE=turbo`, `FLM_ASR=1` kept. The env file now exports `FLM_CONFIG_PATH` / `FLM_XCLBIN_PATH` — required, or flm treats the tag as unknown and silently serves llama3.2:1b. `qwen3.6-moe:35b-a3b` files stay installed for rollback. pi client entry (`~/.pi/agent/models.json` on dankware): id `qwythos:9b`, ctx 262144, input `[text]` (pack has no vision weights), compat `supportsDeveloperRole: false` kept.
## Parallel Streams (measured 2026-08-16)
The NPU serializes generations. It runs exactly one stream at a time. Extra streams queue in FIFO order. FLM server flags `--socket` (default 10) and `--q-len` (default 10) cap queue depth only. They do not add concurrency.
Concurrent-request test, gpt-oss-20b, 128-token cap per stream:
| Streams | Wall s | Aggregate t/s | Per-stream wall s |
|--------:|-------:|--------------:|-------------------|
| 1 | 10.6 | 12.1 | 10.6 |
| 2 | 21.1 | 12.0 | 10.8 / 21.1 |
| 4 | 41.1 | 11.6 | 10.9 / 21.6 / 32.4 / 41.1 |
The 4-stream wall times form a near-perfect 10.7 s ladder. Each stream self-reports 19.3 t/s decode, counting only its own active slices. Trust per-stream wall time, not the self-reported speed, under concurrency.
Consequence: the NPU endpoint is a single lane. It suits low-frequency background work (triage, summarization, agent side-tasks). It does not suit interactive chat for several simultaneous users. The 27B llama.cpp service, by contrast, runs PARALLEL=2 with true concurrent slots.
## Test Runbook
This runbook targets a retired endpoint; keep it as the procedure if the NPU is
ever revived.
**Host RAM availability.** The gate is half of system RAM by default - a relic of the
install: the endpoint ran on a machine that presented 30 GiB of system RAM; on today's
box that ceiling is far larger (live figure: [[Dankclaw System Specs]]). Check free RAM
first, or stop
the service the way the bench policy requires (`sudo systemctl stop
frank-dragoon && sudo systemctl mask frank-dragoon` — self-healing would
otherwise restart it). The NPU shares the unified pool with everything else on
the box.
```mermaid
graph TD
A[1. Raise memlock to unlimited] --> B[2. flm validate + xrt-smi examine]
B --> C{NPU visible to XRT?}
C -- no --> D[Fix driver or firmware path]
C -- yes --> E[3. flm run qwen3.5:4b smoke test, /verbose]
E --> F{Generates tokens?}
F -- no --> D
F -- yes --> G[4. flm serve on a scratch port, never 7770]
G --> H[5. Bench decode + prefill t/s]
H --> I{Speed acceptable?}
I -- no --> J[Record results, stop]
I -- yes --> K[6. Attempt gpt-oss:20b]
K --> L{Loads in ~15 GiB?}
L -- no --> J
L -- yes --> M[Bench, record]
M --> N[7. Last: qwen3.6-moe:35b-a3b, service stopped]
N --> O[Record, decision]
```
1. Raise memlock. Add `* soft memlock unlimited` and `* hard memlock unlimited` to `/etc/security/limits.conf`. Then reboot. For a systemd unit, set `LimitMEMLOCK=infinity` instead.
2. Verify the NPU. Run `flm validate`. Run `xrt-smi examine`. Both must show the NPU. `flm validate` uses the kernel device. `flm run` uses XRT. A passing validate does not guarantee XRT sees the device.
3. Smoke test. Run `flm run qwen3.5:4b`. Toggle `/verbose` for performance reporting. Type `/bye` to exit.
4. Server mode. Run `flm serve qwen3.5:4b`. The default port is 52625. Use a scratch port if 52625 is taken. Never 7770.
5. Bench. Measure decode and prefill t/s at 1k/4k/16k starting context. Record to this page.
6. Step up in size: 9B, then gpt-oss-20b.
7. Attempt `qwen3.6-moe:35b-a3b` only after the smaller runs succeed, and only with the production service stopped. Watch `free -h` during the model download and load.
## Registering packs absent from the catalog (NPU2)
Use when adding a non-catalog NPU2 pack (an extraction or fine-tuned model)
to the FLM registry. Registry facts: user registry
`/home/dank/.config/flm/model_list.json`, shape
`{"model_path": "models", "models": {"<tag>": {"<size>": {name, url,
file_url, size, flm_min_version, files[], vlm, default_context_length,
max_prefill_len, details{format:"NPU2", family, …}, label[]}}}}`; serve
resolves tags through `FLM_CONFIG_PATH` at startup, and `flm list` with the
env var set lists custom tags (`⏬` = not downloaded).
1. Back up the registry JSON first (`cp -a`).
2. Insert the entry via a python json edit + atomic `os.replace`. Set
`files` to the FULL HF file list — **including `vision_weight.q4nx` even
for text-only packs**: the qwen3.5-family loader opens it unconditionally
and a missing file makes the first request HANG silently (no error log).
3. **Do not use `flm pull <tag>` for custom entries** — it throws
`json.exception out_of_range key '<tag>:<size>' not found` while building
the download list. Place files manually into
`/home/dank/.config/flm/models/<PackName>/` with
`curl -sfL "https://huggingface.co/<repo>/resolve/main/<file>?download=true"`
— plain `raw/main/<file>` returns 135-byte LFS pointer stubs, not
weights; verify byte counts against the HF tree API before trusting a
download.
4. Kernel packs: copy from the same-shape family pack (shape-match
`config.json` on hidden_size/num_hidden_layers/head_dim first) into
`/opt/fastflowlm/share/flm/xclbins/<PackName>/` via `sudo cp -a`
(root-owned dir).
5. Deploy/restart the serve unit (the deploy path, not ad-hoc restarts) so
the new process loads with all files present.
6. Verify with a REAL completion request (`model: "<tag>:<size>"`) —
`/v1/models` alone proves registration, not loadability.
Hard constraints: only ONE `flm serve` process can hold the NPU hardware
context — a second concurrent serve process fails every load with
`DRM_IOCTL_AMDXDNA_CREATE_HWCTX err=-22` (aie2 mgmt msg opcode 0x2 status
0x2000003); never run scratch/verify instances while production holds the
NPU. Tag switching happens INSIDE one serve process; the first request
after a model switch pays ~6–11 s cold-load cost (visible in
`usage.load_duration`). Over ssh, never `pkill -f "<pattern>"` with a
pattern present in your own command line — use bracket-guarded patterns
(`[q]wen3.5:4b`) or exact PIDs.
## Tag load-failure diagnosis (200 with an error body)
`POST /v1/chat/completions` answers **HTTP 200** with a flat
`{"error":"Failed to load <tag> model!"}` and no `choices`; `/v1/completions`
returns a nested OpenAI-shaped `server_error` (code 500) also under HTTP
200. A status-code-only client reads this as a successful empty completion —
**always check for the `error` key**. Two distinct causes, distinguished via
`journalctl -u <flm-service> --full`:
1. **Unknown tag** (not in the registry): immediate error, nothing loaded.
Correct behavior, not a bug.
2. **Registered tag whose kernel pack is missing**: FLM resolves the tag,
downloads missing weight files on the fly (first request hangs ~90–110 s),
then fails `Failed to load model: No such file
'/home/dank/.config/flm/xclbins/<PackName>/<kernel>.xclbin'`; subsequent
attempts fail fast (~2 s). Root cause: the serve script exports
`FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins` (user dir), so every
kernel lookup resolves under the user dir, which only contains symlinks
someone linked. **A tag listed in `/v1/models` is NOT proof it is
loadable on this host.** Fix: one symlink per missing pack
(`ln -s /opt/fastflowlm/share/flm/xclbins/<PackName>
/home/dank/.config/flm/xclbins/<PackName>`) — verified live: after the
symlink the first request loaded and answered in 7.4 s wall.
Switch-cost behavior: the `model` field routes per-request (no restart
needed); a request whose model differs from the resident one triggers a full
reload (7–11 s wall for an 8-token completion on ~4B-class packs);
alternating tags pays the swap each time. Single-lane NPU: concurrent
requests queue FIFO behind the current lock holder (`NPU Locked!` →
`NPU Lock Released!`). Probe recipe: `/v1/models` for registered tags; grep
the journal for `Failed to load model` (the quoted xclbin path names the
missing pack dir exactly); diff `ls ~/.config/flm/xclbins/` (symlinks =
installed) against `/opt/fastflowlm/share/flm/xclbins/` (available).
## Incident: qwen3.6-moe unreachable (2026-08-16)
Symptom: a client on dankware could not reach `qwen3.6-moe:35b-a3b` on port 52625. The journal shows three separate causes in the 14:19-15:03 window.
1. **EXEC 203 crash loop (14:19-14:20).** The exec bit was lost on `serve-flm-small.sh` on the target. The unit failed `status=203/EXEC: Permission denied` four times, then started after the bit was restored. The git mode is `100755` (commit `3f42210`), and a `git pull` restores the bit. Verify with `git ls-files -s scripts/serve-flm-small.sh`.
2. **Wrong model ID (14:29-14:52).** The client config still used the model ID `frank-zealot` (a service alias). FLM rejects that tag with `[ERROR] Model tag 'frank-zealot' is not supported.` Each such request also triggered the unknown-tag fallback below.
3. **Swap churn (14:48-15:03).** The swap `gemma4-it:e4b` → `qwen3.6-moe:35b-a3b` restarted the service several times. Each load of the 23 GB model takes about 1.5-2.5 minutes at 262144 ctx. The endpoint does not serve during a load.
Fix: the client model ID must be the FLM registry tag `qwen3.6-moe:35b-a3b`. The working pi agent entry in `~/.pi/agent/models.json` is `baseUrl http://192.168.0.143:52625/v1`, `id qwen3.6-moe:35b-a3b`, `contextWindow 262144`, `reasoning true`, `input [text, image]`. Verified end-to-end from dankware (192.168.0.226) at 15:32 on 2026-08-16.
### Unknown-tag fallback (FLM behavior)
When a request names a tag that is not in the FLM registry, FLM does not return a clean 404. It logs the ERROR, then falls back to the default model `llama3.2:1b`:
- If the `llama3.2:1b` files are missing, FLM downloads them (1.2 GB on 2026-08-16) while the request hangs.
- FLM then loads the 1B model in place of the requested model.
- The files now exist in `~/.config/flm/models/Llama-3.2-1B-NPU2` (1.3 GB) on dankclaw. A future unknown-tag request loads the 1B model silently, with no download and no error.
**Correction (live-probed 2026-08-30):** the fallback no longer fires on the
current serve path — an unknown tag now answers HTTP 200 with a flat
`{"error":"Failed to load <tag> model!"}` body and nothing is loaded or served;
`/v1/completions` answers a nested `server_error` (code 500) under HTTP 200.
Keep client model IDs as FLM registry tags. Never send a service alias. Optional cleanup: move `~/.config/flm/models/Llama-3.2-1B-NPU2` (an incident artifact) to `.AGENTTRASH` so a stray tag re-downloads loudly instead of serving the wrong model.
### Request compatibility: the `developer` role (found 2026-08-16)
pi sessions failed against this endpoint with `Stream ended without finish_reason` while curl worked. Root cause, verified by field bisection:
- The FLM Qwen chat template accepts only the roles `system`, `user`, and `assistant`. Any other role raises `Unexpected message role.` in the template.
- FLM reports the error differently per mode. Non-streaming: HTTP 500 with the template traceback. Streaming: HTTP 200, and the error JSON is the first body. A streaming client sees a stream with no chunks and no `finish_reason`.
- pi sends `role: "developer"` when `model.reasoning && compat.supportsDeveloperRole` (pi source, `openai-completions.js`). The default `supportsDeveloperRole` is true for standard OpenAI-compatible providers, so a reasoning model without a `compat` block always triggers the bug.
- Fix: `"compat": { "supportsDeveloperRole": false }` on the model entry in `~/.pi/agent/models.json`. pi then sends `role: "system"`. Verified with a headless pi session: full round trip, `finish_reason: stop`.
- FLM ignores unknown body fields. `reasoning_effort`, `store`, `max_completion_tokens`, and `tools: []` all arrive in pi requests and pass through without error.
## Risks and Open Questions
- **Ubuntu 26.04 is unofficial.** The distro list stops at 25.10. Validation passes, but breakage would fall on the "generic Linux" path.
- **The 30 GiB budget is untested at the very top.** gpt-oss-20b (15 GiB resident) works. 35B-A3B (~20 GB weights) is still an unknown. Vendor data covers 32 GB and 96 GB boxes only.
- **The FLM serve process was not under systemd** (2026-08-era risk). A reboot
of Dankclaw killed it. The fleet law today is the opposite: every deployed
container/service ships a boot-persistent, self-healing unit, and the retired
NPU unit is deliberately `masked` (`frank-npu.service`, read 2026-10-18) so it
cannot come back at boot.
- **gpt-oss reasoning budget.** The model can spend its full token budget on `reasoning_content` and return empty content. Clients must handle or cap this.
- **No llama.cpp GGUF path.** FLM uses its own pre-compiled kernels. Our UD-Q8_K_XL and HauhauCS builds are not importable. NPU work means FLM-specific model tags.
- **NPU production role (2026-08-16, retired 2026-09-11).** `frank-zealot` then
served `qwen3.6-moe:35b-a3b` on port 52625; the endpoint was retired with the
`amd_iommu=off` decision that day and the unit is masked today. The remaining
NPU content (benchmarks, runbook steps) is research on a device that is again
usable since 2026-10-18 but serves nothing.
## Citations
1. ROCm/FastFlowLM README (chip support, Linux support date, ROCm move, quick start, model storage paths): https://github.com/ROCm/FastFlowLM , accessed 2026-08-16.
2. FastFlowLM docs overview: https://fastflowlm.com/docs/ , accessed 2026-08-16.
3. FastFlowLM Qwen model cards (tags, quantization, context lengths): https://fastflowlm.com/docs/models/qwen/ , accessed 2026-08-16.
4. FastFlowLM gpt-oss model card (15.1 GB weight size, ~15.6 GB NPU cap, 32 GB inconsistency note): https://fastflowlm.com/docs/models/gpt-oss/ , accessed 2026-08-16.
5. FastFlowLM Qwen3.5 benchmarks (v0.9.38, Kraken Point 32 GB): https://fastflowlm.com/docs/benchmarks/qwen3.5_results/ , accessed 2026-08-16.
6. FastFlowLM Qwen3.6 benchmarks (v0.9.45, Kraken Point 96 GB): https://fastflowlm.com/docs/benchmarks/qwen3.6_results/ , accessed 2026-08-16.
7. AMD blog, "FastFlowLM Joins AMD to Advance AI Inference" (2026-07-17): https://www.amd.com/en/blogs/2026/fastflowlm-joins-amd-to-advance-ai-inference.html
8. Wikipedia, "Zen 5" article, Strix Halo section (Ryzen AI MAX+ 395: XDNA2 NPU, 50 TOPS, 256-bit LPDDR5X) via local ZIM `wikipedia_en_all_maxi_2026-02`.
9. Live system checks on Dankclaw, 2026-08-16: `flm validate` output, `dpkg -l` package list, kernel 7.0.0-29-generic, `/proc/3052096/fd` device fd inspection.
serverAppConfigs-docs/LLM/FreeToken Builds.md
-97
View File
@@ -1,97 +0,0 @@
---
type: reference
title: FreeToken Builds
description: Building FreeToken (the official franksdank fork of FreeTok Tokenworks' freetoken vLLM-MTP runtime) on Dankclaw — the CUDA vLLM venv build and the ROCm build, with the dependency traps each one hit.
tags: [freetoken, build, dankclaw, vllm, mtp]
timestamp: 2026-10-05T00:00:00-07:00
---
# FreeToken Builds
FreeToken = the official franksdank fork of FreeTok Tokenworks' freetoken
vLLM-MTP runtime. Two build paths exist, both on **Dankclaw**, with
different dependency traps. Both were verified end-to-end 2026-09-11/12.
## Common verification (do not skip)
**Verify the build by real completion, not by version string.** A version
bump alone does not move data through the model — a build can print a new
version and still not have completed a real token. Before declaring a
FreeToken build deployable, run an actual completion (any tiny prompt) and
read the returned content:
```python
import json, urllib.request
req = urllib.request.Request("http://<host>:<port>/v1/chat/completions",
json.dumps({"model": "frank-zealot", "messages": [{"role": "user",
"content": "Hi"}]}).encode(), {"Content-Type": "application/json"})
print(json.load(urllib.request.urlopen(req, timeout=600)))
```
A 200 with a real `choices[0].message.content` is the only build green.
## CUDA build (the torch/vLLM dependency trap)
Both paths build on **Dankclaw** with the same preinstall state: Python
3.12.10 in the system venv and **Torch 2.10.0.dev** (nightly, repo
download.pytorch.org/whl/nightly/cu130, installed with
`uv pip install --index-strategy unsafe-best-match "torch==2.10.0.dev<date>+cu130"
--index-url https://download.pytorch.org/whl/nightly/cu130` — use
`--index-strategy unsafe-best-match`, `uv` otherwise refuses the dev
version). Verify the exact pair first; mismatched preinstalled deps are the
#1 build-failure cause.
Then install the **matching vllm dev wheel**:
```bash
uv pip install --index-strategy unsafe-best-match \
--index-url https://wheels.vllm.ai/<commit-sha>/vllm-0.6.6.dev0+g<shortsha>.cu130-cp312-cp312-linux_x86_64.whl \
--extra-index-url https://wheels.vllm.ai/nightly/cu130
```
The wheel URL pattern is fixed:
`https://wheels.vllm.ai/<full-commit>/vllm-0.6.6.dev0+g<8-char>.cu130-cp312-cp312-linux_x86_64.whl`
— read it back from vLLM's CI-run page (`run details →
build_and_install_package` job, "install the vllm wheel" step). **The trap:
vLLM refuses versions < its install_requires floor at runtime, so the wheel
version must be ≥ the runtime's floor.** A wheel built at a lower base
(0.5.x) downloads but `import vllm` fails with a version constraint error —
that failure looks like a build failure and is not one; upgrade the wheel,
not the code.
If `uv pip install -e . --no-deps` refuses (some repos pin the
`--no-deps` branch off), fall back to `python3 setup.py` inside the venv,
after `python -m venv --copies .venv && source .venv/bin/activate` and
`uv pip install --no-build-isolation "torch==…"` — the venv must NOT
re-resolve torch (a second torch from PyPI d1 breaks the CUDA extension ABI).
## ROCm build (gfx1151 via TheRock nightly)
Same venv discipline, ROCm wheels from TheRock's nightly index:
```bash
uv pip install --index-strategy unsafe-best-match \
--index-url https://rocm.nightlies.amd.com/v2/gfx1151-dcgpu-supports_fbsgens_split_0717-rc1/ \
"torch==2.10.0.dev<date>+rocm" "triton==3.6.0" "pytorch-triton==3.6.0"
```
Then `vllm` itself (`uv pip install -e . --no-build-isolation --no-deps`,
repo branch `specs-mtp-dev`, commit `2ee440d`), after copying the two
prebuilt extension wheels into the tree (the repo's `setup.py` skips
compiling them when found). The ROCm build also needs
`install-fastrak-gfx1151.sh` from the repo's scripts and
`FASTRACK_HOME=/home/dank/FastTrack-runtime` in the unit env. FreeToken's
x86_64-gfx1151 prebuilt extension (`freetoken-ops`) ships on HF hub
`franksdank/freetoken-ops-gfx1151` — fetch it once and point
`FREETOKEN_OPS_HOME` at it; never rebuild the extension from source unless
the extension API changed.
`--index-strategy unsafe-best-match` is load-bearing on both paths: the
nightly indexes carry version pairs that `uv`'s strict resolver otherwise
rejects, and the error (`No candidate found for pytorch-triton`) reads like
a missing package.
## Related
- [[Model Swap and Rollback]] — the swap/deploy story these builds serve
- [[Llama Cpp Serving Sizing]] — the harness measurement discipline
serverAppConfigs-docs/LLM/GGUF Compatibility.md
+8 -8
View File
@@ -1,7 +1,7 @@
---
type: reference
title: GGUF Compatibility and Runtime Lock-in
description: How to answer "will engine X load this GGUF?" from the file's header alone, why a custom tensor type forces the runtime, and the discipline for pinning a fork build that can actually serve the file.
description: "Answering whether an engine loads a GGUF from the file header alone, and pinning a fork build that can."
tags: [gguf, llama.cpp, forks, quantization, speculative]
timestamp: 2026-10-05T00:00:00-07:00
---
@@ -14,7 +14,7 @@ Studio refuse custom types at load (`unknown tensor type`); a fork-private type
means exactly one engine can serve the file. Measured on this fleet repeatedly:
`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is ftype 115 with 311 tensors of type 103
(`GGML_TYPE_Q8_0_ROCMFPX`) — only charlie12345/ROCmFPX reads it. The full type
tables live in [[Llama Cpp Runtimes]] and [[Llama Cpp Code Diff]]
tables live in and
(ggml_type 100–108, ftype families 100–106 / 110–113 / 118).
## 1. Read the header, do not guess
@@ -69,14 +69,14 @@ filenames lie about quantization and about MTP/vision heads.
```
Validate before trusting: the parse is right iff the tensor-info block ends
somewhere sane and names look real (`blk.0.attn_q.weight`,
somewhere sane and names look real (`blk.0.attn_q.weight`
`token_embd.weight`, `output_norm.weight`). EOF throw = wrong enum or
offset — do not work around it.
3. **Diff what engines actually bind on**: `general.architecture`; tensor
count; tensor name list in order; per-tensor `ggml_type` and dims; geometry
KVs (`<arch>.block_count`, `.embedding_length`, `.feed_forward_length`,
`.attention.head_count[_kv]`, `.rope.*`, `.ssm.*`, `.context_length`,
KVs (`<arch>.block_count`, `.embedding_length`, `.feed_forward_length`
`.attention.head_count[_kv]`, `.rope.*`, `.ssm.*`, `.context_length`
`.nextn_predict_layers`); MTP tensors (`nextn` in the name); vision tensors
(`visual`, `v_` prefixes). A same-family fine-tune quantized by the same
tool usually comes out byte-identical in every one of these — that is the
@@ -117,8 +117,8 @@ re-download.
## Related
- [[Llama Cpp Runtimes]] — which fork ships which custom types
- [[Llama Cpp Code Diff]] — the enum tables and code-level differences
* [[kingjones-q8agent-rocmfpx]] — the era that served the ftype-115 file, and
- which fork ships which custom types
- the enum tables and code-level differences
* the era that served the ftype-115 file, and
[[frank-dragoon]] for what runs there now
- [[Llama Cpp Serving Sizing]] — pricing what a payload costs once it loads
serverAppConfigs-docs/LLM/Heterogeneous Split Triage.md
+8 -8
View File
@@ -1,7 +1,7 @@
---
type: reference
title: Heterogeneous Split Triage
description: Triage before building when someone asks to split a model across unequal accelerators on one host (24 GB GDDR6X + LPDDR APU, PCIe x4 link) — link characterisation, the real bottleneck, what placement never buys, UMA pools, drafter splits, and the verdict ranking.
description: "Triage before splitting across unequal accelerators: link characterisation, real bottleneck, what placement never buys."
tags: [heterogeneous, multi-device, llama-cpp, triage, pcie]
timestamp: 2026-10-05T00:00:00-07:00
---
@@ -93,7 +93,7 @@ of bytes), so it is tempting. Before proposing it:
(compare GGUF `tokenizer.ggml.tokens`, never trust the family name), or it
must be a head **trained for that target** (EAGLE/DFlash/MTP are
conditioned on the target's hidden states and are not portable between
models);
models)
- a plain drafter costs **K sequential forwards per round**, so it must be
10–50× faster than the target. Break-even with `r = t_draft/t_target`:
`rate_ratio = A/(K·r + 1)`, `A = (1−p^(K+1))/(1−p)`; at `r ≈ 0.4–0.5`
@@ -107,10 +107,10 @@ of bytes), so it is tempting. Before proposing it:
(draft gets its own device list) plus `-ngld`/`-otd`/`-ctkd -ctvd`.
The llama.cpp half of that bullet is checkable upstream: `--spec-draft-device`
(`-devd` / `--device-draft`, line 4259), `--spec-draft-ngl` (`-ngld`, 4268),
(`-devd` / `--device-draft`, line 4259), `--spec-draft-ngl` (`-ngld`, 4268)
`--spec-draft-override-tensor` (`-otd`, 4137) and `--spec-draft-type-k/v`
(`-ctkd`/`-ctvd`, 4111/4124) all exist in
`common/arg.cpp` ([source](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/common/arg.cpp),
`common/arg.cpp` ([source](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/common/arg.cpp)
read 2026-10-18).
## 6. Expressing a split (llama.cpp, the usual vehicle)
@@ -122,8 +122,8 @@ read 2026-10-18).
- buffer types match by name at load (`ggml_backend_buft_name`), so device
names work as `-ot` values;
- `-cmoe`/`-ncmoe` are **CPU-only** (`--cpu-moe` / `--n-cpu-moe`,
`common/arg.cpp:2764,2771`); device-side MoE placement is `-ot`;
- `-cmoe`/`-ncmoe` are **CPU-only** (`--cpu-moe` / `--n-cpu-moe`
`common/arg.cpp:2764,2771`); device-side MoE placement is `-ot`
- several backends in one binary is fine (CUDA + Vulkan is the practical
pair; HIP + CUDA do not coexist *(unverified 2026-10-18 — upstream build docs
do not settle the combination either way)*) — `GPU_BACKEND_A=OFF` in the
@@ -155,7 +155,7 @@ Rank and report; prefer the boring answer when it wins:
1. **Two independent endpoints** (zero shared traffic) — often the real
answer to "what's a useful combination", especially over a narrow link.
2. **One process, weights-resident, KV co-located with attention on the
fast pool** — promising and quantifiable (+% decode at long context);
fast pool** — promising and quantifiable (+% decode at long context)
needs a dual-backend build and a baseline measurement.
3. **Hot-expert cache / per-token weight streaming** — affordable only if
the miss set per token is small (a few experts; check
@@ -169,5 +169,5 @@ Rank and report; prefer the boring answer when it wins:
## Related
- [[PCIe Link and Atomics]] — the link and collective walls underneath
- [[Llama Cpp Multi-Device Splits]] — the deployed split shapes
- the deployed split shapes
- [[Llama Cpp Serving Sizing]] — KV and pool arithmetic
serverAppConfigs-docs/LLM/Hybrid CPU GPU Processing Dankserver.md
-225
View File
@@ -1,225 +0,0 @@
---
type: reference
description: "Dankserver uses NVIDIA GPU with system RAM for hybrid processing."
tags:
- llm
timestamp: 2026-08-28
---
# Hybrid CPU GPU Processing Dankserver
## Overview
Dankserver has **no NVIDIA GPU**: the RTX 3090 left the box on 2026-09-15 and the host now runs two AMD Radeon AI PRO R9700s (`lspci | grep -ci nvidia` → 0, read 2026-10-18 — [[Dankserver System Specs]]). This page's vLLM-era hybrid guidance is reference material, not the serving path: the live service is radiance 1.0.13 as `frank-archon`, and its hybrid mechanism is a pinned host pool (43.88 GiB per rank, `scripts/frank-archon.env:184`) plus the CPU-resident 47.68 GiB n-gram/PLE table, not vLLM KV swapping ([[frank-archon]]).
## Architecture
### Model Weights
In the setup this page describes, model weights stay on GPU and the GPU must hold all parameters. As a statement about vLLM it is no longer true: current vLLM ships CPU weight offload as `cpu_offload_gb`, plus the grouped prefetch path (`offload_group_size`) — `vllm/config/offload.py:23,54` ([source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/config/offload.py), read 2026-10-18). What remains true is the capacity arithmetic: offloaded weights cost PCIe traffic per token (see [[PCIe Link and Atomics]] §2).
### KV Cache
KV cache moves between GPU and CPU:
1. Active KV cache pages stay in GPU VRAM
2. Inactive pages swap to CPU RAM
3. Swapped pages write to disk swap space if RAM is full
4. Pages reload to GPU when needed
### Compute Distribution
- GPU handles all matrix multiplication and attention computation
- CPU handles tokenization, detokenization, and input processing
- CPU handles KV cache swap operations
- GPU worker processes run on CPU cores via multiprocessing
## Optimal Settings for Dankserver
### GPU Memory Utilization
Set `--gpu-memory-utilization` based on your GPU:
| GPU | Recommended Value |
|-----|-------------------|
| 24 GB (RTX 3090/4090) | 0.7-0.8 |
| 40 GB (A100) | 0.8-0.9 |
| 80 GB (A100 80GB) | 0.85-0.95 |
| 80 GB (H100) | 0.9-0.95 |
Dankserver has neither NVIDIA card today: it carries 2× 32 GB (32,624 MiB) R9700s.
The live engine does not use this knob at all — it claims an explicit per-card
pair instead (`--vram-weights-mib 25600` / `--vram-kv-mib 5120`,
`scripts/frank-archon.env:230-231`).
Lower values leave more GPU memory for KV cache. This reduces swap frequency.
### Swap Space / KV Offload
Current vLLM has no `--swap-space` engine argument, and the CPU KV-offload knob pair
is owned by [[RAM CPU Offloading in VLLM]] (that page carries the upstream
`CacheConfig` verification). In Podman, mount the host cache read-write for KV offload
writes. The table below shows historical `--swap-space` guidance as a sizing reference.
| System RAM | Recommended Swap |
|------------|------------------|
| 32 GB | 8-16 GB |
| 64 GB | 16-32 GB |
| 128 GB | 32-64 GB |
| 256 GB | 64-128 GB |
More swap space allows longer contexts and more concurrent requests.
### Tensor Parallelism
Use tensor parallelism for large models:
```
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 4 \
--gpu-memory-utilization 0.85
```
### KV Cache Data Type
Use FP8 KV cache for Ampere and newer NVIDIA GPUs. On this host's RDNA4 cards the live engine runs an fp8 KV cache too (`kv fp8` — `scripts/frank-archon.env:264`, which also records that this engine takes only bf16|fp8 for the whole cache).
```
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.9
```
FP8 KV cache reduces memory usage by 50% compared to BF16.
## NUMA Optimization
Dankserver reports **one** NUMA node today (`lscpu` → `NUMA node(s): 1`, `NUMA node0 CPU(s): 0-7`, read 2026-10-18), so NUMA binding buys nothing on this host; these flags matter on multi-node topologies:
```
vllm serve model \
--tensor-parallel-size 4 \
--numa-bind
```
This pins each GPU worker to its nearest NUMA node. This reduces memory access latency.
For explicit NUMA binding:
```
vllm serve model \
--tensor-parallel-size 4 \
--numa-bind \
--numa-bind-nodes 0 0 1 1
```
## CPU Resource Requirements
VLLM needs CPU cores for:
- 1 API server process
- 1 engine core process
- N GPU worker processes (one per GPU)
Minimum physical cores = 2 + N (where N is GPU count).
For Dankserver, which has **2** GPUs (2× R9700) and 8 cores (`lscpu`, read
2026-10-18):
- Minimum 4 physical CPU cores by the 2 + N rule
- Recommended 8+ physical CPU cores — the host has exactly 8 (SMT off)
## Performance Tuning
### Chunked Prefill
Chunked prefill is enabled by default. Tune with:
```
vllm serve model \
--max-num-batched-tokens 8192
```
Higher values improve throughput. Lower values improve latency.
### CUDA Graphs
CUDA graphs improve throughput. Disable for faster startup:
```
vllm serve model \
--enforce-eager
```
### Compilation Cache
Reuse torch.compile artifacts:
```
export VLLM_CACHE_ROOT=~/.cache/vllm
vllm serve model
```
Copy cache between boots for faster startup.
### Skip Memory Profiling
Skip memory profiling on repeated boots:
```
vllm serve model \
--kv-cache-memory <value_from_first_boot_log>
```
## Monitoring
Check GPU memory:
```
rocm-smi
```
Check for preemption warnings in VLLM logs.
Check CPU utilization:
```
htop
```
## Example Commands
### Single GPU, Medium Model
```
vllm serve Qwen/Qwen3.6-35B-A3B \
--gpu-memory-utilization 0.85 \
--max-num-batched-tokens 4096 \
--max-num-seqs 128
```
### Single GPU, FP8 Model
```
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.9 \
--max-num-batched-tokens 8192
```
### Multi-GPU, Large Model
```
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 4 \
--gpu-memory-utilization 0.9 \
--max-model-len 8192
```
## See Also
- [[VLLM Overview]]
- [[VLLM Index]]
- [[RAM CPU Offloading in VLLM]]
- [[Shared Memory Architecture Dankclaw]]
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
serverAppConfigs-docs/LLM/LLM Bench Windows and Measurement.md
-271
View File
@@ -1,271 +0,0 @@
---
type: reference
title: LLM Bench Windows and Measurement
description: Stopping-and-restoring benchmark windows on the fleet LLM hosts (llama-bench sweeps, HTTP param benches, multi-arm vLLM offload sweeps), honest benchmark methodology, and the usage-counter measurement law under speculative decoding.
tags: [benchmarking, llama-bench, ab-testing, speculative, vllm, fleet]
timestamp: 2026-10-05T00:00:00-07:00
---
# LLM Bench Windows and Measurement
Fleet law first: **STOP the production service before any GPU experiment**
(AGENTS.md Benchmarking Policy) — a bench beside a live service starves the
GPU and corrupts results; an unisolated GPU experiment has precedent for
wedging a host until power cycle. Reference harnesses:
`scripts/bench-batch-ubatch.sh`, `scripts/autoresearch.sh`,
`scripts/autoresearch/frank-archon-arms.sh`.
## 1. The window structure (one config per arm, a driver owns the window)
1. **Stop + gate before every arm** — `wait_for_clear`: unit NOT active AND
no RUNNING container of that name (`podman ps`, not `-a`; `-a` counts
EXITED containers and can never clear — `rm -f` the exited one first) AND
port free (`ss -ltn`). The graceful-stop race cost a whole arm 2026-10-01:
`ExecStop=podman stop -t 120` drains up to 2 min; a driver that sleeps 3 s
then polls `/health` gets **200 from the still-running PRODUCTION engine**
— the arm "passes" with production's boot log, then `podman exec` dies
with `container state improper`.
2. **Boot the arm as the service, not a scratch port** (no scratch ports on
LLM hosts — one card, one server). The serve script takes its config via
env-file override (`FRANK_ZEALOT_ENV_FILE`, `FRANK_ARCHON_AR_ENV_FILE`)
pointing at a staged copy; the checkout is never mutated.
3. **Fast-fail the health wait**: poll `podman ps | grep -qx <name>` inside
the loop — a refused boot dies in one poll instead of 20 min. Capture the
serve script's FULL stdout on failure (a `tail -8` loses engine-init root
causes; raise happens above the tail).
4. **Measure** (§4), capture boot-log engine facts (KV pool, arena slots,
SpecDecoding lines) — they survive measure crashes.
5. **Teardown + converge**: kill by explicit PID/pgid (never pattern kills),
`podman rm -f` scratch, wait `MemAvailable ≥ ~230 GiB` on dankserver
before the next boot; restore production from the EXIT trap and verify the
**conjunction oracle**: health 200 + `is-active` + `is-enabled` +
mask-symlink clean. A bare `systemctl start` is not a restore oracle.
(`systemctl mask` FAILS on a relinked unit — log it, explicit stop is
enough: systemd never auto-restarts an explicitly stopped unit.)
6. **Warm boots cycle ~2 min** (page cache), cold ~8 min — size timeouts for
mixed speeds.
**Destructive tails (2026-10-01):** a scratch script whose tail does
`systemctl stop` + `podman rm -f <PRODUCTION container>` fires later if a
probe loop survived the driver kill — it can destroy a just-deployed
production engine (signature: deploy exits green, then seconds-to-minutes
later `Exited (0)`, `Deactivated successfully`, clean exit = removed, not
crashed). Before ANY deploy: `pgrep -af` every scratch script of the session
and kill strays by explicit PID; killing the dev-side job does NOT kill the
host-side `deploy-remote.sh` (two of them once managed the same unit).
Two failure shapes inside the window that read as "the bench died":
- **Stale reads from a stopping container.** `podman stop` answers while the
engine inside is still flushing; a `podman logs` taken in that window
returns the boot-prefix only. Poll `podman ps` for `Exited (0)` (or
`podman wait`) before reading logs, or you will write "no results" into
the table for a run that has them.
- **A D-state worker wedges the stop.** A bench child that wedged in the
kernel ignores SIGTERM, so `podman stop -t N` burns its full timeout and
then the container is `Stopping` forever. Kill by container first
(`podman kill <ctr>` → SIGKILL path), verify the process is gone from
`podman ps` AND `nvidia-smi`, and only then re-serve. Never let the
window script proceed to a redeploy with a zombie bench worker holding
VRAM.
## 2. llama-bench sweeps (offline batch/ubatch/split/KV/depth)
**What llama-bench cannot express — decide BEFORE designing:** no
`--ctx-size` (`n_ctx = n_prompt + n_gen + n_depth`, so a bench can NEVER prove
activation buffers fit beside the DEPLOYED KV — only the real unit answers
that), no `--parallel` (no slot shape, no concurrency), no `--spec-*`
(`tg` numbers are the drafterless baseline, useful as a control only), and a
warmup run is itself a full-size run. `-d N` is a depth PREFIX (paid once per
cell via a `llama_state_seq_*` save/restore); without it a `tg` run is at
~zero depth and meaningless for long conversations.
**The separator trap:** `/` separates devices WITHIN one configuration, `,`
separates sweep values — for both `-dev` and `-ts`. `-ts 36,29` parses as the
single value 36 → everything on one card. `--tensor-split 36,29` (server) →
`-ts 36/29` (bench). Ranges exist: `-ub 512-2048+256`.
**Harness shape:** one invocation per cell (an infeasible cell calls
`exit(1)` and kills every later cell of a cross-product invocation — and
infeasibility is usually the question); TSV row per cell as it completes +
`bench.json` (`-o json`) + `bench.log` (`--progress`) + a 2 s `rocm-smi
--showmeminfo vram` VRAM trace per cell (a cell that allocated less than
intended looks fast); resume by skipping non-empty cell outputs; stop the
service in the harness, restore from `trap … EXIT`, verify afterwards. Match
the deployment's load flags exactly (`-ngl -sm -ts -fa -ctk/-ctv -t`).
**Run under a transient unit, not a shell:**
```bash
sudo systemd-run --unit=<name> --collect --no-block --property=Type=oneshot \
--property=LimitMEMLOCK=infinity --property=User=dank \
/home/dank/serverAppConfigs/scripts/<harness>.sh
```
Deploy units set `LimitMEMLOCK=infinity` for ROCm staging; an SSH shell has
8 MiB and fails the load for an unrelated reason. This also survives ssh
disconnects. **Boot rungs under the production environment, not a bench
shell** — a rootless shell probe booted 3×262144 bf16 that production refused
(systemd `User=` + memlock change what the pinned-expert path holds):
`sudo systemd-run --uid=<user> --unit=<name> -p LimitMEMLOCK=infinity -p
Restart=no env VARFILE=… /path/serve.sh`.
**Verify before burning hours:** dry-run against a STUB binary with
`SERVICE=<nonexistent-name>` (the trap/stop never reached), tiny grid,
including one forced-`exit 3` cell — prove the TSV row, failure capture,
resume logic. Validate the real flag set in ~1 s by pointing `-m` at a
nonexistent file. **Reading:** bracket the sweep (reference cell first AND
last); check per-cell `stddev_ts` (a difference smaller than spread is not a
result); single-sequence decode is usually flat across batch/ubatch (prefill
knob); batch ≥ ubatch (llama.cpp clamps ubatch to batch, silently duplicating
smaller-batch cells); report what the grid cannot prove (drafter, slots, real
context).
## 3. HTTP benches of server-side parameters
`llama-bench` benches the bench binary, not the running server — server-side
params (`--threads-batch`, `--ubatch-size`, `--spec-type`, `--parallel`) need
HTTP:
- **Prefill**: large prompt, `max_tokens=1` (server crashes on 0), measure
`prompt_tokens / wall_time`.
- **Decode**: `max_tokens=16384` minimum, NEVER 1 (warmup, cache growth and
spec acceptance all stabilize over thousands of tokens); measure
`predicted_per_second` from `timings` or `completion_tokens / wall_time`;
temperature 0.
Known fleet harnesses: `scripts/bench-threads-batch-http.py`,
`scripts/bench-decode-threads-batch.py`, `scripts/bench-spec-decode.py`.
sglang-specific: `max_tokens=1` on `/v1/chat/completions` with the qwen3
reasoning parser streams zero deltas — use `/v1/completions`.
## 4. The measurement law (under speculative decoding this is absolute)
**Never count SSE chunks.** Rejected drafts break the 1:1 chunk↔token
mapping; chunk counting reported 8.3 tok/s for a config whose server counters
said 23–26 — a 3x false regression that nearly abandoned the change that
worked. Always:
```python
decode_tok_s = usage.completion_tokens / (wall - ttft)
# stream=True, stream_options: {include_usage: true}; ttft = first content-bearing chunk
```
vLLM's 10 s `Accepted throughput` log lines are a rolling window, not a
measurement (6.2 → 13.1 across adjacent windows on identical settings) — use
them for acceptance STATISTICS only (`Avg Draft acceptance rate`, `Mean
acceptance length`, per-position rates). Warm pass + ≥3 runs, median;
temperature 0; ≥256 tokens. **A decode "jump" with early-stopped completions
(ct < 256) and acceptance > 90 % is degenerate generation** (the fork's
repetition-halt fired at ~140 tokens on corrupted numerics) — inspect text
before believing a win. Run-to-run spread is 5–10 %: repeat arms; one sample
nearly shipped the wrong setting.
## 5. Honest methodology
- **Match the corpus to the workload.** Token soup under-reports speculative
decoding (2.9–3.6 tok/update vs 3.86–5.8 on real code) and mis-ranks draft
settings; acceptance, not step time, moves throughput. Build the corpus
from the repo's own sources: `{"prompt": "<instruction>\n\n<file body>"}`,
`--custom-output-len`.
- **Salt every run** with a unique per-arm token in the first bytes — it is
the prefix-cache key (and a proxy routing key). Without it arm 2 measures
arm 1's warm cache.
- **Two workload shapes need two clients:** distinct prompts (batch
behaviour, prefill parallelism) and sequential turns with growing history
(TTFT of the first streamed chunk per turn — the only measurement that
exposes prefix-cache behaviour). These pull in opposite directions: a cold
batch spreads prefill across replicas, a conversation's next turn only pays
if it lands where its prefix is cached.
- **Read the engine's ledger, not the card:** `Available KV cache memory` /
`GPU KV cache size: N tokens` / `Maximum concurrency for M tokens per
request: X.Xx` is the capacity number. vLLM's `Prefix cache hit rate:` line
is a LIFETIME average across every prompt the replica ever saw — send the
same prompt twice to an idle engine and compare TTFTs (measured 22.3 s cold
→ 1.4 s warm for a 50k prefix). `GPU KV cache usage` says the pool is not
full; it does NOT say a prefix survived. The KV budget is NOT monotonic in
chunk size (5.44 GiB at CHUNK 8192, 4.22 at 16384, 4.16 at 4096) — measure
three points before treating a knob as a lever.
- **Soak before believing:** 15–30 min of concurrent families, count failures
AND latency; report hit/miss split and medians with min/max — the median of
a bimodal distribution describes neither mode.
- **Verification that is not the benchmark:** run the client the fleet
actually uses; and a deployed unit is not proven until an unattended reboot
brings it back.
- **Record the "no" answers** with numbers and mechanism — placement balance,
concurrency admission and chunk-size tweaks all measured flat or worse
here; the wall was arithmetic (cached prefixes + live sequences share one
pool).
## 6. Sweep hygiene (the arms that produced wrong verdicts)
- **Prefetch uniformity:** every arm in a sweep carries the same
expert-prefetch setting. The 2026-09-29 campaign measured losing arms with
prefetch ON and incumbents OFF; untaxed, verdicts inverted (no-spec
12.11 → 41.47; gather 32×64 parity → +18.7 %). Never re-read a taxed
verdict.
- **Knob-landed preflight:** for every `-e` knob, confirm
`podman exec <ctr> printenv <VAR>` — a knob the fork silently ignores
produces a "parity" arm that is really the default (v1 of the gather sweep
lost all 3 arms this way; serve-script hooks are additive and can drop
vars).
- **Always run an explicit OFF arm:** the 2026-09-29 campaign's only win was
deleting a baseline's silent `--expert-prefetch 1` (+40–51 %). Challengers
compare against default-off, not "incumbent + one flag".
- **Pin the capacity bar:** `--kv-cache-memory-bytes` (contract) rather than
autofit (remainder) so pool numbers are reproducible; compute need =
tokens × measured B/token (hybrid models: KV dtype does NOT double mamba
states — measure, don't assume 2x).
- **Report decode AND prefill (TTFT) per rung:** a decode-only view nearly
deployed a 2x-prefill-tax config. Per-depth tables, never a single median —
one fork's gather win grows with depth (54 t/s @1k, 39.5 @short).
- Container recreate (`podman rm -f`) wipes `podman cp`'d harnesses — re-copy
`mtp_measure.py`/`bench_iter.py` after EVERY recreate, or mount them.
## 7. Bash laws that ate windows (each cost an arm)
1. `${VAR:+SPEC=$1}` in front of a command expands to a bare word →
`SPEC=3: command not found`, nothing runs. Pass literal assignments:
`SPEC="${1:-}" bash serve.sh`; the callee's `${SPEC:+--flag}` skips empty.
2. `${2WAY:-0}` is fatal `bad substitution` (digit-leading name), and inside
an if/else it aborts ONLY that compound statement — the script CONTINUES,
every arm tears down clean with zero measurements. Detection law: `ls`
the per-arm `.measure.txt` files before trusting any summary; arm echoes
are not evidence.
3. `local a=$1 b=$x` expands `$x` before the builtin assigns — under `set -u`
it aborts AFTER the harness stopped production. Two statements.
4. `sed -i` renames the file; already-running bash keeps the OLD inode —
mid-flight patches miss arms launched in the preceding minutes. Patch
early or run a make-up window.
5. Never heredoc python through ssh→zsh→bash — write to a file, ship it
(`ssh host 'cat > /path && chmod +x /path' < local.sh`); nested quoting
fails silently. Same trap in `git commit -m`: backticks become command
substitution.
6. No comments INSIDE a backslash-continued command (the continuation joins
the comment); verify multi-line command construction with `bash -n` AND a
render test.
## 8. Results location and recovery (frank-archon arms)
| copy | path |
|---|---|
| per-arm evidence | `$ARMS_DIR` = `~/.jobs/<run-stamp>/` (`.rc`, `.done` beside it) |
| summary | `$ARMS_DIR/summary.tsv` = `arm rc decode_tps decode_tps_t0.7 kv_tokens boot_s` + `<arm>.out/.env/.overlay/.rc/.metrics` |
| server-side last window | `/home/dank/.autoresearch-frank-archon/run.json` |
| durable memo | `.../last/run.json` + `last/hash` (promoted only after bench AND verified restore) |
| fallback | `~/.omp/agent/sessions/<ws>/<session>/<N>.{shake,bash}.log` (permanent) |
`/tmp/frank-archon-arms-*` and `/tmp/armstub/` are stub self-tests —
constants 99.9/88.8 identical across arms by construction; **identical
numbers across arms == stub or bug, never a measurement.** After compaction,
re-harvest with `grep -rhoE 'decode_tps[a-z0-9_.]*=[0-9.]+'` over the session
logs. A window costs ~20 min warm; a detached runner's timeout ≥ 3600 s per
window.
## Related
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan arithmetic being swept
- [[DFlash DSpark Speculative Decoding]] — spec A/B protocol and shapes
- [[LLM Performance Diagnosis]] — profiling and attribution methods
- [[Remote Job Runner]] — detached execution of long sweeps
- [[frank-medium-llamacpp-dflash2]] — the measured ubatch-optimum history
serverAppConfigs-docs/LLM/LLM Performance Diagnosis.md
+8 -9
View File
@@ -1,7 +1,7 @@
---
type: reference
title: LLM Performance Diagnosis
description: Probing why a served LLM is slow — the flat-depth probe, the expert-offload duty-cycle arithmetic, kernel-time attribution via the profiler window, live telemetry without benchmarking, RAM-bandwidth ceilings, IBS tensor attribution, prefix-cache-aware routing, and the step-shape census behind decode stalls under a prefill.
description: "Probing why a served model is slow: the flat-depth probe, offload duty-cycle arithmetic, and step-shape attribution."
tags: [performance, profiling, moe, pcie, prefix-cache, routing, telemetry]
timestamp: 2026-10-19T00:00:00-07:00
---
@@ -21,7 +21,7 @@ the cost; the problem is per-token expert movement and KV/attention tuning is
ruled out entirely. A genuinely context-bound server shows a clear downward
slope. Record cold vs warm separately — a small gap means the cache is
working. The canonical flat run (fp8 expert-offload shape, a few hundred to
sixteen thousand tokens of context) is in [[Tcclaviger Fork Ops]]'s profile.
sixteen thousand tokens of context) is in 's profile.
## 2. Duty-cycle arithmetic (the "GPU not fully utilized" answer)
@@ -86,7 +86,7 @@ Worked example (GLM-5.3-Flash, 2×GPU, offload, decode 9.9 tok/s):
No amount of link arithmetic would have produced those two conclusions: the
suspect was 1 %, and the mass was pointwise glue from an uncompiled trunk —
whose own boot log names the cause (`FULL_DECODE_ONLY` override). It also
whose own boot log names the cause (`FULL_DECODE_ONLY` override). It
predicted a negative result in advance (spec decode cannot help a shape whose
per-token cost scales with the token count) which matched an unexplained flat
MTP measurement.
@@ -117,7 +117,7 @@ per-request telemetry — it measures the real workload and costs nothing.
- One `max_tokens=200` request is production traffic and fine; anything that
pins the card is a bench → stop-and-mask first. Check WHICH instance you
are reading (`systemctl status | head -12`; task ids jump across
restarts). The engine's own ledger (`KV self size`, `n_slots`,
restarts). The engine's own ledger (`KV self size`, `n_slots`
`n_ctx_slot`) outranks your model of it.
## 5. The fork's data path (before blaming the link)
@@ -130,7 +130,7 @@ tokens, so the step can silently cross the threshold onto the slow path as
draft depth grows. Check the threshold's premise: a default tuned at
12.4 MiB/expert does not transfer to 2.344 MiB/expert, where copy-in can be
several times faster. TunableOp ghost-skip: `CLAV_TUNABLEOP_SWEEP=1` alone
does NOTHING under expert offload (`CLAV_TUNABLEOP_WITH_OFFLOAD=1` required);
does NOTHING under expert offload (`CLAV_TUNABLEOP_WITH_OFFLOAD=1` required)
check for the skip line before concluding autotuning ran. Gather-geometry
knobs read at import time → each arm is a full reload.
@@ -189,7 +189,7 @@ analyze; `--analyze-only`).
Measure with a **sequential-turn client**, not only a batch benchmark — a
batch of distinct prompts parallelises prefill across replicas and hides the
problem. The two failure modes: round-robin loses every other cache hit
(90k-token contexts: warm-turn TTFT 2676 ms round-robin vs 1096 ms pinned);
(90k-token contexts: warm-turn TTFT 2676 ms round-robin vs 1096 ms pinned)
pinning-everything loses a card (16 requests: 18.6 s spread vs 30.1 s
pinned). **The rule that gets both: route by request shape** — a request
carrying more than one message is a continuation (agent loop re-sending its
@@ -209,7 +209,7 @@ is not a per-request verdict (probe directly: same prompt twice to an idle
replica — 22.3 s cold → 1.4 s warm); salt corpora; record sample sizes;
`vllm bench serve` needs `--tokenizer <local path>`; verify metric names
against the live endpoint before dashboards. Test shape: drive the routing
function directly with synthetic bodies (stdlib-only policy module);
function directly with synthetic bodies (stdlib-only policy module)
conversation identity is the FIRST N KiB of prompt — a shorter tested prefix
hashes differently per turn and pinning silently never happens.
@@ -281,8 +281,7 @@ Run on: [[frank-archon]] (radiance 1.0.13, TP2 `expert_tiered`, budget 2048 = in
## Related
- [[LLM Bench Windows and Measurement]] — the window discipline and usage-counter law
- the window discipline and usage-counter law
- [[PCIe Link and Atomics]] — link probing and the streaming-vs-compute ceiling
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan behind duty-cycle terms
- [[vLLM Porting and Compile Triage]] — the compile fix for the pointwise-glue bound
- [[frank-archon]] — the endpoint where §9's step-shape census was run
serverAppConfigs-docs/LLM/Llama Cpp Multi-Device Splits.md
-287
View File
@@ -1,287 +0,0 @@
---
type: reference
title: Llama Cpp Multi-Device Splits
description: What llama.cpp can and cannot express across devices, the no-collective layer-split path, split balancing by measurement, drafter KV sizing from GGUF headers, and heterogeneous placement triage.
tags: [llama.cpp, multi-gpu, layer-split, pcie, sizing, speculative]
timestamp: 2026-10-05T00:00:00-07:00
---
# Llama Cpp Multi-Device Splits
## 1. Expressibility law — classify the arrangement before running anything
llama.cpp expresses exactly two things across devices, and they are not
interchangeable:
| arrangement | how | expressible? |
|---|---|---|
| **Layer split** — layers 0..n on dev A, rest on dev B | `-sm layer -ts <a>/<b>` | **yes**, no collectives needed; works cross-vendor (CUDA + Vulkan measured 2026-09-16) |
| **Per-tensor placement** — attention+KV on A, experts on B | `-ot <regex>=<dev>` | **NO.** `--override-tensor` sets `has_tensor_overrides()`, a REQUIRED term of the scheduler's `pipeline_parallel` flag (only read site: `llama-context.cpp`, in the `pipeline_parallel = …` expression). That flag sets `n_copies = parallel ? MAX : 1` in `ggml_backend_sched_new`; with `n_copies == 1` a pre-allocated weight on a non-primary device has no backend and the graph aborts at `ggml-backend.cpp:941` (`pre-allocated tensor (blk.0.ffn_down_exps.weight) in a buffer (Vulkan1) that cannot run the operation (NONE)` — `(NONE)` is the leaf's own op name). |
Upstream anchors, re-read 2026-10-18 against `master`:
`src/llama-context.cpp:517-526` computes `pipeline_parallel` with
`!model.has_tensor_overrides()` as a required term;
`ggml/src/ggml-backend.cpp:1089` carries the verbatim abort
`pre-allocated tensor (%s) in a buffer (%s) that cannot run the operation (%s)`
(the `:941` anchor above is where it sat in the `37b53fd` build this was
measured on); and `common/arg.cpp` defines `-sm/--split-mode` (2821),
`-fit/--fit` (2882), `--spec-type` (4295), `--spec-draft-device` (4259),
`--spec-draft-n-max` (4161), `-fa/--flash-attn` (1758) — every flag in §4 is
real upstream, not fork-only
([source](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/common/arg.cpp)).
Failure modes observed (mainline `37b53fd`): per-tensor `-ot exps=Vulkan1`
with `-dev CUDA0,Vulkan1` → scheduler abort; layer-scoped `-ot` with a
cross-backend pair → model load fails with only `failed to load model`;
`-ot x=CPU` **works** because the CPU backend is last in the list — **the CPU
is the only legitimate partner in a heterogeneous split.** On a CUDA+AMD host
a card can be the trunk OR the offload host, never the complement of an
AMD-resident peer. Pure Vulkan changes nothing here — the gate is the
override, not the driver mix.
If the plan needs tensors on a specific device: stop, the answer is no for
any vendor mix. Expressible alternatives, in order: (1) two independent
endpoints, one model per device; (2) single-vendor layer split; (3) device +
CPU per-tensor offload (mainline pinning guard: `GGML_CUDA_NO_PINNED`; ik fork
was `GGML_CUDA_NO_PINNED_WEIGHTS` — without it a CPU-overridden tensor can
land in a pinned `CUDA_Host` buffer and page-lock ~108 GiB).
Layer-split arithmetic to check before promising: a whole layer carries its
experts (MoE) — compute GiB/layer, not per param-group; only some layers carry
KV (every 4th on the hybrid), so a share of layers is only a share of KV; big
non-repeating tensors (PLE tables) land on the main device and cannot move —
they cap how small the primary can get.
## 2. No collectives? The atomics wall and the layer-split answer
Decide with one line before designing:
```bash
sudo lspci -vv -s <card2> | grep AtomicOpsCtl # ReqEn- ⇒ no collectives, ever
lspci -tv # a Routing- bridge is a hardware wall
```
`ReqEn-` kills vLLM entirely — TP needs a per-layer allreduce AND PP needs a
device-backend `send`/`recv` of the hidden state
(`vllm/distributed/parallel_state.py`) — i.e. the same RCCL primitive that
fails on an atomics-less card. llama.cpp's `--split-mode layer` needs no
collective: layers distribute in order, only the boundary activation crosses
PCIe once per forward pass. Full probe/repair detail:
[[PCIe Link and Atomics]].
## 3. Build: pin by feature, bundle the runtime at the exact depth
- Pin the ref by FEATURE list (e.g. DFlash2 + qwen35 MTP on 2026-09-16 →
`ggml-org/llama.cpp#27342`); never float. Verify a fork build the same way
(`ls <venv>/…/spec_decode/ # dflash/ vs dflash2/`).
- `-DCMAKE_HIP_ARCHITECTURES=<gfx only>` — a sibling arch links, loads, then
segfaults. `-DGGML_HIP_RCCL=OFF` when there is no atomics path (only tensor
mode degrades to the butterfly fallback; layer split unaffected).
- **TRAP — `cp -r SRC DST/` flattens when `DST` does not exist** and drops one
directory level. rocBLAS/hipBLASLt look for device code at a FIXED depth
(`../lib/rocblas/library`, `../lib/hipblaslt/library/<arch>`); flattened,
the engine loads every GiB, answers `/health`, and dies on the FIRST request
with `Cannot read .../TensileLibrary.dat: No such file or directory for GPU
arch`. `mkdir -p DST && cp -r SRC/. DST/`, then
`patchelf --set-rpath '$ORIGIN/../lib' build/bin/llama-*`.
- Dual-backend build (CUDA + Vulkan is the practical pair; HIP+CUDA do not
coexist): `-DGGML_CUDA=ON -DGGML_VULKAN=ON` in one tree; Vulkan needs
`glslc` + libvulkan headers. **Device names are not what you expect:**
`--list-devices` lists the discrete card under BOTH backends (`CUDA0` AND
`Vulkan0`) and the APU LAST (`Vulkan1`) — a wrong `-ot` name makes
llama.cpp print the valid set and exit (self-diagnosing).
- What the existing build can do: `GPU_BACKEND_*=OFF` lines in its
CMakeCache.
## 4. Serve flags that matter on a split
```bash
HIP_VISIBLE_DEVICES=0,1 llama-server \
--device ROCm0,ROCm1 --split-mode layer --tensor-split 1,1 \
--gpu-layers 999 --fit off --flash-attn on \
--ctx-size <n> --parallel 1 --cache-type-k f16 --cache-type-v f16 \
--jinja --metrics --cache-prompt \
--spec-type draft-dflash --spec-draft-model <dflash2.gguf> \
--spec-draft-device ROCm1 --spec-draft-n-max 7
```
- `--fit off` + explicit `--ctx-size`: auto-fit must not resize a context set
on purpose. **`--parallel N` DIVIDES `--ctx-size`** — one slot at large ctx
is the concurrency policy, concurrent agent loops queue.
- Pin the drafter to the card holding the target's LAST layers
(`--spec-draft-device`); under tensor mode an unpinned draft joins the
meta-device KV allocation and aborts the KV build. `--spec-draft-n-max` must
be ≤ the drafter's `block_size - 1` (DFlash2: 8 ⇒ 7).
- Qwen3.8-family templates RAISE on `reasoning_effort=high` (500) — probe
low/medium/high/xhigh and map harness levels before declaring ready.
## 5. Split balancing: measure, never interpolate
**A `-ts` step is ONE LAYER, and hybrid stacks do not have uniform layers.**
qwen35 is 16 × (3 Gated DeltaNet + 1 Gated Attention): one step ≈ 0.43 GiB on
a DeltaNet boundary, ≈ 1.43 GiB on a full-attention one (weights + that
layer's own f16 KV) — measured on frank-medium. The gap can jump 2.9 GiB in a
single step; the best integer split can sit 0.2 GiB from balance while its
neighbours are 1+ GiB off. An affine fit reproduces the deployed point and
overstates gaps in between (`tests/frank_medium_unit.bats` encodes the fit as
a gate; treat a NEGATIVE fitted constant as a unit bug).
Knobs that move weight and REQUIRE a re-measure: `--ubatch-size` (compute
buffers are **~2x heavier on the output/drafter card** — 512→1536 added
~3.3 GiB there vs ~1.7 GiB elsewhere), a drafter swap (weights + draft KV),
`--ctx-size`/`--parallel` (KV). Total free VRAM is invariant across splits —
only the distribution moves; use that to sanity-check a suspicious row.
**Procedure (one split per load):**
```bash
WORK=/tmp/split.env; cp scripts/serve-<svc>.env $WORK
export FRANK_MEDIUM_ENV_FILE=$WORK
for ts in 39,26 38,27 37,28 36,29 35,30; do
sed -i "s|^TENSOR_SPLIT=.*|TENSOR_SPLIT=$ts|" $WORK
nohup bash scripts/serve-<svc>.sh >/tmp/split.log 2>&1 &
# poll /health (or process exit), then:
rocm-smi --showmeminfo vram | grep 'VRAM Total Used' # sample ~6 s after health
pkill -x llama-server; sleep 6
done
```
Load → sample → kill, NO requests sent (allocation only). Run under `nohup`
and poll from a separate call (a load is 30–90 s). Pick the smallest
inter-card gap that clears the per-card floor with room to spare; one `-ts`
step ≈ 1.5 GiB at ~6k batch / 262k ctx on 30 GiB cards — enough to bracket,
not to solve. Prefer balance: the busy card's headroom bounds the context
(`ctx_extra ≈ (floor − measured_free) / per-position cost on the busier
card`). A faulted split (`SDMA0 UTCL2 page fault` → `illegal memory access` →
`ggml_abort`) is OFF-LIMITS — re-measure, never re-tune; more headroom does
not prevent a boundary-copy fault.
## 6. Drafter KV: sized from the TARGET, bounded by its window policy
`common/speculative.cpp` builds the draft context with
`cparams.n_ctx = llama_n_ctx(ctx_tgt)` — `common/speculative.cpp:2663` on
`master`, re-read 2026-10-18 — **no flag caps it.** So:
```
draft KV = ctx_tgt × block_count × 2 (K,V) × kv_heads × key_length × dtype
```
…unless the drafter declares a sliding window. Measured at 262,144 target
ctx: DFlash2 (`sliding_window = 2048`) ≈ 40 MiB; a same-shape 5-layer drafter
with NO window = 20 KiB/position = **5.0 GiB** — a four-layer swing in the
balance point, and swapping them invalidated an already-balanced split.
**Failure signature:** the engine loads the whole model, answers `/health`,
and dies on the FIRST draft decode with `ROCm error: out of memory` /
`cudaMalloc failed` inside `common_speculative_impl_draft_dflash::process` —
a sizing failure, not a broken drafter; under a self-healing unit it presents
as a crash-loop. Read `sliding_window`, `block_count`, `head_count_kv`,
`key_length` from the drafter's GGUF header first. A compact reader (the
three format traps: no KV alignment — v3 alignment applies only to the tensor
section; arrays are `type(9) → element_type(u32) → count(u64)`, element type
FIRST; the enum is `7=BOOL 9=ARRAY`, 64-bit = 10/11/12, and 8=STRING/9=ARRAY
is the v2/v3 file order, NOT gguf.h's 2/3):
```python
import struct, sys
def rd(path):
f = open(path, 'rb'); assert f.read(4) == b'GGUF'
struct.unpack('<I', f.read(4)); struct.unpack('<Q', f.read(8))
nk = struct.unpack('<Q', f.read(8))[0]
def s():
n = struct.unpack('<Q', f.read(8))[0]; return f.read(n).decode('utf-8', 'replace')
SCALAR = {0:1,1:1,2:2,3:2,4:4,5:4,6:4,7:1,10:8,11:8,12:8}
FMT = {0:'<B',1:'<b',2:'<H',3:'<h',4:'<I',5:'<i',6:'<f',7:'<?',10:'<Q',11:'<q',12:'<d'}
keys = {}
for _ in range(nk):
k = s(); t = struct.unpack('<I', f.read(4))[0]
if t == 8: keys[k] = s()
elif t in SCALAR: keys[k] = struct.unpack(FMT[t], f.read(SCALAR[t]))[0]
else:
et = struct.unpack('<I', f.read(4))[0]; n = struct.unpack('<Q', f.read(8))[0]
if et == 8: keys[k] = [s() for _ in range(n)]
elif et in SCALAR: keys[k] = ('arr', et, n); f.read(n * SCALAR[et])
else: raise Exception('unsupported arr elem type %d' % et)
return keys
```
## 7. Heterogeneous placement triage (go/no-go with arithmetic)
- **Link:** `LnkCap` vs `LnkSta` at every hop; expect **idle-downshift** (a
Gen4 x4 dock idles at Gen1 x4, ~1 GB/s) — measure under load. Per-hop cost
vs hidden state (5120-dim f16 = 10 KiB/hop ≈ 1.3 µs at Gen4 x4): bytes are
almost never the constraint; **hop count is** — every boundary is a host
sync, and a per-op-split runtime can hit 100 splits/token at ~5 % SM. Read
`graph splits` from the loader log.
- **KV is usually the bottleneck:** `KV/token = kv_bytes_per_position ×
context` (attention reads the whole cache every token). Highest-leverage
placement: **KV on the fastest-memory device together with its attention
compute** — never KV alone over the link (GBs over 7.88 GB/s = hundreds of
ms/token). Check fit: `kv_positions × slots × bytes + attention_trunk ≤
VRAM`.
- **What placement never buys:** prefill is compute-bound and expert weights
are read once per batch — moving them buys nothing and adds a serial stage.
Calibration: Strix Halo + 3090, moving 12.5 % of expert mass to the GPU
gave decode +12 % but prefill 192 → 32 tok/s. The harmonic
`1/(f_big/BW_slow + f_fast/BW_fast)` reproduces the decode gain within ~2
points (assumes sequential stages — true at batch 1).
- **UMA check before assuming additivity:** `mem_info_vram_total` (BIOS
carve-out) vs `mem_info_gtt_total` (aperture onto SYSTEM RAM). A big GTT
window means the "GPU memory" IS the same LPDDR: adding a stage there adds
a compute engine, not capacity/bandwidth. Only discrete VRAM is additive.
- **Speculation as the "cheap" split:** bandwidth-free (token ids per round)
but needs the exact tokenizer for a plain drafter or a head trained for
that target; a plain drafter costs K sequential forwards (break-even needs
p ≈ 0.7+ at r≈0.4–0.5, K=3); compare against the INCUMBENT (a target
already committing ~1.7 tok/round on its own MTP head is a high bar); and
check the runtime accepts external drafters at all (closed engines 400
unknown drafter names by design). Details: [[DFlash DSpark Speculative Decoding]].
- **Hazards:** keep paged/sidecar tables where they are read from; keep mmap
ON for huge-embedding models (anonymous memory → cgroup kill); CUDA graph
capture can abort on unusual graph shapes; make the FIRST measurement a
single-device baseline of the runtime on the new device — a split's
arithmetic is worthless if the baseline loses to the incumbent; a split
consumes the fast device entirely — check what it was serving.
**Verdict ranking:** (1) two independent endpoints — often the real answer;
(2) one process, weights-resident, KV co-located with attention on the fast
pool; (3) hot-expert cache only if `misses × bytes × tok/s` fits the link and
overlaps — no mainstream runtime implements it; (4) name what does NOT help:
TP across unequal devices, KV on the far device, loading over the link,
prefill-motivated placements.
## 8. Harness syntax and verification
`llama-bench` separates a LIST with `/` and SWEEP values with `,`:
`-dev CUDA0,Vulkan1` is TWO single-device runs, never one two-device run;
`-ts 0.85,0.15` is two sweep values each becoming `[0.85, 0, …]` — device 0
takes everything. Write `-dev A/B -ts x/y`. Read the table's own `dev` column;
any placement-dependent number needs an independent memory check
(`nvidia-smi` sampling while loaded). (This artifact produced a wrong fleet
conclusion 2026-09-16 — "CUDA + Vulkan cannot serve one model" — with two
bogus OOMs that were really all-111-GiB-on-the-24-GiB-card.)
Verification battery: `/health`; both cards' memory ≈ half the weights each
(busier card also carries the drafter; compare `model_bytes/2 + drafter +
ctx×KV/2 + buffers`); raise ctx by N and watch BOTH cards rise by
N·bytes/2 — neither card's total may equal the model file (that arithmetic
rules out replication); `spec_decode_num_drafts_total` > 0 and a real
completion; a deep prompt (140k) proves a raised window is REAL, not
declared; repeat prompt → `timings.prompt_n` collapses (prefix cache live);
`/slots` shows expected `(id, n_ctx)` pairs; two concurrent requests with
`journalctl -o cat | grep 'slot print_timing'` proves drafting on BOTH slots
(default journal output truncates these lines).
Ops traps: a server started over ssh outlives stopping the runner (keeps the
port — `pkill -x <binary>` first); `systemctl start` of a root unit needs
sudo (polkit "Access denied" otherwise); old crash `ROCm error` lines live in
the journal forever — check `ActiveEnterTimestamp`/`NRestarts` before
treating a grep count as a live fault.
## Related
- [[PCIe Link and Atomics]] — the atomics wall, link probing, repair
- [[Llama Cpp Serving Sizing]] — slot-pool and KV pricing
- [[DFlash DSpark Speculative Decoding]] — spec types, chaining, A/B
- [[frank-medium-llamacpp-dflash2]] — the layer-split era runbook
- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — the measured history
serverAppConfigs-docs/LLM/Llama Cpp Serving Sizing.md
+8 -9
View File
@@ -1,7 +1,7 @@
---
type: reference
title: Llama Cpp Serving Sizing
description: Slot-pool sizing against a headroom band, per-device VRAM accounting, prompt-cache/state sizing and the QSA flash-attn fix — the measurement-first procedures for llama.cpp serving shapes.
description: "Measurement-first sizing for llama.cpp shapes: slot pools against a headroom band, per-device accounting, prompt cache."
tags: [llama.cpp, vram, kv-cache, slots, flash-attn, sizing]
timestamp: 2026-10-05T00:00:00-07:00
---
@@ -19,7 +19,7 @@ guesswork.
`n_ctx_slot = CTX_SIZE / PARALLEL`. A hybrid model with
`kv_unified = 'false'` gives each slot its own cache AND a fixed per-slot
state buffer — slot count is a memory term, not free.
2. **Two boots on the real card** (production stopped, per the bench law),
2. **Two boots on the real card** (production stopped, per the bench law)
each near the target; record `nvidia-smi` used/total plus the boot log's
`n_slots`/`n_ctx_slot`: shape A = current/cheap, shape B = new demand.
`non-KV = used − (tokens × KiB/token)`; the two rows separate
@@ -31,7 +31,7 @@ guesswork.
slot gets an equal integer window. Boot and measure; if outside the band,
walk one step — never re-derive from a formula.
4. **Propagate the per-slot number — it is the client contract:**
`models.yml` `contextWindow` = `n_ctx_slot` (NOT the model's native window),
`models.yml` `contextWindow` = `n_ctx_slot` (NOT the model's native window)
or long conversations die with `400 exceed_context_size_error` instead of
compacting; compaction/summary budgets sized to the old window get
re-checked; endpoint docs and the rollback table move too.
@@ -66,13 +66,13 @@ co-tenants.
- **Engine ledger first.** Halogen prints weights pinned (+ remaining 2 MiB
contiguous blocks — the honest fragmentation signal: a block census that drops
across a restart means the model did not come back into the same hole; the
before/after census this was learned on is that era's, on [[frank-dragoon]]),
before/after census this was learned on is that era's, on [[frank-dragoon]])
device budget, KV pool/working split, "left for everything
else". llama.cpp prints them at `llama-model.cpp:1827` (per-backend model buffer)
and `llama-context.cpp:692` (compute buffer) on the era's build; on today's
`master` the same `LLAMA_LOG_INFO` lines sit at `llama-model.cpp:2009`
("model buffer size =") and `llama-context.cpp:842` ("compute buffer size =")
([source](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/src/llama-context.cpp),
([source](https://raw.githubusercontent.com/ggml-org/llama.cpp/master/src/llama-context.cpp)
re-read 2026-10-18) — but **`llama-bench`
suppresses loader INFO and a systemd unit did not put them in the journal**
(verified `grep -c` = 0). Recipe: one launch through `llama-cli`/`llama-server`
@@ -92,7 +92,6 @@ co-tenants.
projector, never weights alone. Do
the capacity subtraction before designing, on the card number the owner page
quotes, not the advertised one.
- **Context lengths: derive, never remember.** `llama-bench` computes
`n_ctx = n_prompt + n_gen + n_depth` (defaults 512/128/0, type_k/v F16,
flash_attn AUTO, split_mode LAYER). `-p 512,16384 -n 64` = contexts
576/16,448; a plain `tg` row has ~zero depth — `-d` is the only thing that
@@ -116,7 +115,7 @@ of the whole sequence + constant recurrent state + the draft context's KV:
`per_token = 2 × n_full_attn_layers × n_kv_heads × head_dim × bytes (K and V) +
draft_per_token`. The worked qwen35-hybrid instance of that formula (full layer
count, f16 KV, MTP draft) and the slot-size table it prices are on
[[frank-medium-llamacpp-dflash2]]; cross-check any such arithmetic against the
; cross-check any such arithmetic against the
journal's `prompt state size` lines — the two must agree.
**The journal is the instrument:** `exceeds cache size limit … skipping` =
@@ -171,6 +170,6 @@ part of that era's record on [[frank-dragoon]].
- [[GGUF Compatibility]] — what a payload is before it is priced
- [[frank-zealot]] — the endpoint whose Eras rows are the two-boot priced shapes this method produced
- [[frank-medium-llamacpp-dflash2]] — the state-law worked example behind section 3
- the state-law worked example behind section 3
- [[DFlash DSpark Speculative Decoding]] — the spec A/B protocol on llama.cpp
- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — the multi-device side
- / — the multi-device side
serverAppConfigs-docs/LLM/Model Swap and Rollback.md
+10 -10
View File
@@ -1,7 +1,7 @@
---
type: reference
title: Model Swap and Rollback
description: The full model-swap discipline for the LLM fleet — fit preflight, fetch and payload verification, runtime build/flag checks, git-only config, live validation, systemd handoff, close-out, the three-pin revert law, and research-verification discipline.
description: "The model-swap discipline: fit preflight, fetch verification, build and flag checks, live validation, three-pin revert."
tags: [models, swap, gguf, huggingface, rollback, deployment, verification]
timestamp: 2026-10-05T00:00:00-07:00
---
@@ -51,9 +51,9 @@ express; per-tensor cross-device placement stays refused under
## 2. Ground truth first
- Read the HF model card (runtime pin, hard constraints, recommended args)
+ the repo trio `scripts/<svc>.env`, `scripts/serve-<svc>.sh`,
+ the repo trio `scripts/<svc>.env`, `scripts/serve-<svc>.sh`
`scripts/<svc>.service` + the runbook under `serverAppConfigs-docs/LLM/`.
- Check the host before downloading anything: `ls /home/dank/models/`,
- Check the host before downloading anything: `ls /home/dank/models/`
`df -h /home`, `du -sh` — bench-era or prior-deployment files often already
exist.
@@ -129,7 +129,7 @@ arms possible — keep it intact.
`images` is silently ignored; probe with ≥256 px).
- vLLM specifics: first request after boot can take ~18 s (deferred Triton
loads) — warm and discard; `--reasoning-parser` / tool parsers are part of
the contract ([[vLLM Boot Forensics and Config Interrogation]]).
the contract ().
## 7. systemd handoff — THE TRAP
@@ -205,7 +205,7 @@ Pure measurement sessions that changed no state need only pin 1.
ladder UD-IQ1_S 93.1 GB → UD-Q4_K_XL 199.7 GB, GSQ-RCO 3.5-bit −1.40 pp
MMLU-Pro vs Q8_0).
- GGUF parsing gotcha: expert tensors are
`blk.N.ffn_{gate,up,down}_exps.weight` — 4 dot-parts INCLUDING `.weight`;
`blk.N.ffn_{gate,up,down}_exps.weight` — 4 dot-parts INCLUDING `.weight`
a `len(parts) == 3` filter matches zero layers with no parse error. Filter
on `parts[0]=="blk" and parts[3]=="weight" and "exps" in parts[2]`.
@@ -214,7 +214,7 @@ Pure measurement sessions that changed no state need only pin 1.
When a host must serve two models at once (an iGPU runtime plus a second
card), or one endpoint is moving and a new one takes the freed hardware.
**0. Decide the shape before touching hardware.** One provider per
**0. Decide the shape before touching hardware.** One provider
HOST:PORT in `~/.omp/agent/models.yml` (on the box where the harness runs)
— a second endpoint is a new provider block, not a second model in the
existing one. Read the budget as an inequality: for unified memory the
@@ -235,7 +235,7 @@ suffix — and check what pointed at them: bench harnesses, a chat template
only the old runtime read, and `scripts/test-changed.sh` mappings are the
usual victims (a mapping to a retired test file fails the pre-commit
wrapper silently for weeks). One bats file per service asserting
INVARIANTS (`pool == 2 × ctx`, KV bytes ≤ card budget, port separation),
INVARIANTS (`pool == 2 × ctx`, KV bytes ≤ card budget, port separation)
not constants; a negative grep must strip comments first
(`grep -vE '^\s*#' file | grep -q …`) or it matches the prose explaining
the very thing you removed.
@@ -258,12 +258,12 @@ target — including whether `/metrics` answers — is [[frank-dragoon]]'s wire
contract.
**4. Cutover, then verify by driving it.** `deploy.sh <target> --service
<large> --service <small>` (order matters); per endpoint: `/health`,
<large> --service <small>` (order matters); per endpoint: `/health`
`/v1/models` reports the alias, a chat completion returns content AND
`reasoning_content`, a `tools` request returns `tool_calls`, an image
request is answered (or refused with a named reason), the runtime's own
speculation counters move, and the unit is `enabled` and `active`. All three LLM
endpoints are **root** units today (`scripts/frank-archon.service`,
endpoints are **root** units today (`scripts/frank-archon.service`
`scripts/frank-dragoon.service`, `scripts/frank-zealot.service`), so linger is
not what gates them; linger matters for rootless quadlets. Read the engine's boot
ledger for a silent clamp. Finally spend one
@@ -282,6 +282,6 @@ predecessor "stopped" (the port race above).
- [[GGUF Compatibility]] — which runtime can load a payload at all
- [[Llama Cpp Serving Sizing]] — pricing the shape once it loads
- [[RDNA4 MoE Expert Offload VRAM Budget]] — vLLM-side fit arithmetic
- [[LLM Bench Windows and Measurement]] — measuring the new deployment
- measuring the new deployment
* [[frank-archon]] — the live Dankserver endpoint; the tcclaviger vLLM era
(2026-09-29 → 10-05) is a row in its Eras table
serverAppConfigs-docs/LLM/PCIe Link and Atomics.md
+22 -22
View File
@@ -1,7 +1,7 @@
---
type: reference
title: PCIe Link and Atomics
description: Link probing (idle downshift, path ceilings, CUDA/HIP bandwidth probes), the streaming-vs-compute ceiling arithmetic, and the PCIe-atomics wall that decides whether AMD multi-GPU collectives can exist at all.
description: "Link probing, the streaming-versus-compute ceiling arithmetic, and the atomics wall that decides multi-GPU collectives."
tags: [pcie, rccl, atomics, multi-gpu, bandwidth, amd, nvidia]
timestamp: 2026-10-05T00:00:00-07:00
---
@@ -61,13 +61,13 @@ H2D rate`; `ceiling = 1 / link_time × resident-fraction saved (LRU hits)`.
Compare against the compute-in-RAM path at host RAM bandwidth (256 GB/s
LPDDR5X on the Strix Halo — 128 GB LPDDR5x-8000 on a 256-bit bus,
[amd.com spec page](https://www.amd.com/en/products/processors/laptop/ryzen/ai-300-series/amd-ryzen-ai-max-plus-395.html)).
**If the link is >10x slower than the memory behind it (x4 is 31x here),
**If the link is >10x slower than the memory behind it (x4 is 31x here)
streaming loses by construction and no tuning rescues it; doubling the link
only doubles a losing number.**
Worked example: the Dankclaw Gen4 x4 run of 2026-09-16 — measured H2D/D2H
rates, the 3 MiB slab cost, and the streaming-vs-compute comparison it decided
— is recorded on [[x4-pair-experiment]] §The link, measured. Read it once and
— is recorded on §The link, measured. Read it once and
apply the rule above: on that link the streaming design lost by 3–5x, and the
x8 ceiling would only have halved the loss.
@@ -88,7 +88,7 @@ time, so encode them:
+ `hipSetDevice` only. First line: `setvbuf(stdout, NULL, _IONBF, 0);` — a
segfaulting process loses block-buffered stdout and hides the crash point.
- Build with an explicit arch: `HIP_TARGETS=gfx1201 hipcc -O2
--offload-arch=gfx1201 -o p p.hip`; compile with `| grep -E "error:"`,
--offload-arch=gfx1201 -o p p.hip`; compile with `| grep -E "error:"`
never `| tail -N` (truncation hid a missing `#include <cstring>`).
- `hipMemset`/`hipDeviceSynchronize` are `[nodiscard]` — use or cast.
- Mount the whole `/dev/dri` into ROCm containers; individual `renderD*`
@@ -109,28 +109,28 @@ static double one(size_t bytes, bool pinned, int iters) {
if (hipMalloc(&dev, bytes) != hipSuccess) return -1;
if (pinned) { if (hipHostMalloc(&host, bytes, hipHostMallocDefault) != hipSuccess) return -1; }
else { host = malloc(bytes); }
memset(host, 0x5a, bytes);
hipEvent_t a, b; hipEventCreate(&a); hipEventCreate(&b);
for (int i = 0; i < 3; i++) hipMemcpy(dev, host, bytes, hipMemcpyHostToDevice);
hipDeviceSynchronize();
hipEventRecord(a, 0);
for (int i = 0; i < iters; i++) hipMemcpy(dev, host, bytes, hipMemcpyHostToDevice);
hipEventRecord(b, 0); hipEventSynchronize(b);
float ms = 0; hipEventElapsedTime(&ms, a, b);
memset(host, 0x5a, bytes)
hipEvent_t a, b; hipEventCreate(&a); hipEventCreate(&b)
for (int i = 0; i < 3; i++) hipMemcpy(dev, host, bytes, hipMemcpyHostToDevice)
hipDeviceSynchronize()
hipEventRecord(a, 0)
for (int i = 0; i < iters; i++) hipMemcpy(dev, host, bytes, hipMemcpyHostToDevice)
hipEventRecord(b, 0); hipEventSynchronize(b)
float ms = 0; hipEventElapsedTime(&ms, a, b)
double gb = (double)bytes * iters / (ms / 1000.0) / 1e9;
if (!pinned) free(host); else hipHostFree(host);
hipFree(dev); hipEventDestroy(a); hipEventDestroy(b);
if (!pinned) free(host); else hipHostFree(host)
hipFree(dev); hipEventDestroy(a); hipEventDestroy(b)
return gb;
}
int main(int argc, char **argv) {
setvbuf(stdout, NULL, _IONBF, 0);
setvbuf(stdout, NULL, _IONBF, 0)
if (hipSetDevice((argc > 1) ? atoi(argv[1]) : 0) != hipSuccess) return 1;
size_t sz[] = {2459136UL, 3UL<<20, 16UL<<20, 64UL<<20, 256UL<<20, 1UL<<30};
for (int i = 0; i < 6; i++) {
size_t b = sz[i];
int iters = (b >= (1UL<<30)) ? 10 : (b >= (256UL<<20)) ? 20 : 100;
printf(" %10.3f MiB | pinned %6.2f | pageable %6.2f\n",
b/1048576.0, one(b,true,iters), one(b,false,iters));
b/1048576.0, one(b,true,iters), one(b,false,iters))
}
return 0;
}
@@ -185,7 +185,7 @@ sudo lspci -vv -s <bridge> | grep AtomicOpsCap # want Routing+
slot on CPU root ports (atomics fine), the rest behind the chipset (usually
not). Bandwidth for TP is small anyway: ~10 MB per token-step for a 27B at
batch 4 is a millisecond-scale figure at x4 — compute-bound, not
interconnect-bound (full derivation: [[Dual-R9700-Cooperation]] §Bandwidth).
interconnect-bound (full derivation: §Bandwidth).
**Probe matrix (bare 2-rank `torch.distributed`, one process per GPU, 25 s PG
timeout) — one probe answers several design questions:**
@@ -216,11 +216,11 @@ reporting an outage (`gitea` vs `gitea-server`).
complex; x4 suffices for TP.
2. Board whose CPU x16 bifurcates x8/x8 (done 2026-09-20 on dankserver — the
per-device chain census, including both endpoints reading `ReqEn+`, is
[[Dual-R9700-Cooperation]]'s). Re-checked 2026-10-18 without root: both root
's). Re-checked 2026-10-18 without root: both root
ports report `current_link_width=8` at `32.0 GT/s`
(`/sys/bus/pci/devices/0000:00:01.{1,3}/current_link_width`), and the live
engine states direct P2P between the cards.
3. llama.cpp `--split-mode layer` — no collectives at all ([[Llama Cpp Multi-Device Splits]]); costs the GGUF conversion plus paged attention /
3. llama.cpp `--split-mode layer` — no collectives at all (); costs the GGUF conversion plus paged attention /
continuous batching / spec decode.
4. Independent TP=1 replicas + routing proxy — cards never share one model.
@@ -230,7 +230,7 @@ is gated by a CUDA-sm-keyed table (`{9.0, 10.0, 10.3, 10.7}`; ROCm reports
`CUSTOM_ALL_REDUCE_MAX_SIZES` in
`vllm/distributed/device_communicators/all_reduce_utils.py`, whose keys on
today's `main` are exactly `"9.0"`, `"10.0"`, `"10.3"`, `"10.7"`
([source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/distributed/device_communicators/all_reduce_utils.py),
([source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/distributed/device_communicators/all_reduce_utils.py)
read 2026-10-18); nothing keys on a ROCm-reported `12.0`. — and patching it is
insufficient because TP init's `broadcast`/`barrier`/logits `all_gather` have
no non-RCCL path. A
@@ -249,6 +249,6 @@ kernel's declared args (`roc-obj-extract` + `llvm-readelf --notes`), look for
## Related
- [[Llama Cpp Multi-Device Splits]] — the layer-split path around the wall
- [[Dual-R9700-Cooperation]] / [[x4-pair-experiment]] — measured history
- the layer-split path around the wall
- / — measured history
- [[frank-archon]] — the gather economics of the retired tcclaviger vLLM era (2026-09-29 → 10-05)
serverAppConfigs-docs/LLM/Pipeline Parallelism/Ethernet Bandwidth Math.md
-530
View File
@@ -1,530 +0,0 @@
---
type: reference
title: Ethernet Bandwidth Math
description: Bytes-per-token math for pipeline vs tensor parallelism, feasibility tables for 1/2.5/10 GbE, and real multi-node benchmark numbers.
tags:
- pipeline-parallelism
- research
timestamp: 2026-08-17
---
# Bandwidth Analysis: Multi-Node LLM Inference over Ethernet
## 0. Measured on the Qwen3.8-27B PP deployment (2026-08-17)
Everything in this section was measured on the live deployment
([[Qwen3.8 PP Deployment]]) while sampling `/proc/net/dev` at 50 Hz on
both ends. The theoretical tables below use H=4096/L=32; the real model
is larger (see constants), which raises the per-token bytes but not the
conclusions.
### 0.1 Verified constants (from the GGUF + env, not guessed)
| Constant | Value | Source |
| -------- | ----- | ------ |
| `hidden_size` (embedding_length) | 5120 | GGUF metadata (`qwen35.embedding_length`) |
| Blocks | 65 (45 on worker, 20 on head) | `--tensor-split 0.68,0.32` |
| FA layers | 16 total, interval 4 (11 worker / 5 head) | GGUF `full_attention_interval` |
| KV per FA layer per token | 3136 B (K q8_0 1088 + V f16 2048) | env file comment, confirmed |
| ubatch | 256 | `UBATCH_SIZE` in `frank-dragoon.env` |
| MTP verify step positions | 5 (1 + n_max 4) | `SPEC_DRAFT_N_MAX=4` |
| Link (old path) | 1 GbE via USB-C adapter on claw | `enx00e04cc2e1e5`, negotiated 1000 Mb/s |
| Link (current path) | 2.5 GbE eno1 to eno1 | negotiated 2500 Mb/s both ends |
| iperf3, 2.5 GbE path | 281 MB/s claw->server, 294 MB/s server->claw | measured |
| ping RTT, 1 GbE USB path | min 0.239 / avg 0.742 / max 1.085 / mdev 0.334 ms | measured |
| ping RTT, 2.5 GbE path | min 0.224 / avg 0.585 / max 0.883 / mdev 0.211 ms | measured |
The USB-C adapter added ~15 us to RTT (0.239 vs 0.224 ms min) and more
jitter (mdev 0.334 vs 0.211 ms).
### 0.2 Measured wire bytes (the number that matters)
Raw hidden state per position per direction = 5120 x 2 = **10.0 KB**.
The *measured* bytes per position per direction are ~4x that:
| Direction | Measured | Raw hidden |
| --------- | -------- | ---------- |
| head -> worker | 39.8 KB/position | 10.0 KB |
| worker -> head | 38.4 KB/position | 10.0 KB |
The extra ~30 KB/position is the serialized RPC graph (per-position
nodes for all 45 worker blocks: attention, KV append, SSM ops)
plus protocol framing. It scales with position count just like the
payload, so it does not shrink away.
Per ubatch (256 positions) the measured crossing is **20.5 MB**
(10.4 MB head->worker + 10.1 MB worker->head).
### 0.3 Measured throughput and the validated model
Measured prefill (fresh 13,959-token prompt, 14k context, 2.5 GbE
path): **457.9 t/s** (30,485 ms, journal task 11924). Decomposition:
| Term | 2.5 GbE (measured anchor) |
| ---- | ------------------------- |
| wire per ubatch (20.5 MB) | 70 ms |
| compute + overhead per ubatch (residual) | 489 ms |
| ubatch total | 559 ms = 2.18 ms/token |
Using that residual and only swapping the wire time per link speed:
| Link | wire/ubatch | predicted prefill | avg link use | line-rate burst duty |
| ---- | ----------- | ----------------- | ------------ | -------------------- |
| 1 GbE (old USB path) | 174 ms | **386 t/s** | 15.8% | 26.2% |
| 2.5 GbE (current) | 70 ms | 458 t/s (matches measured) | 6.4% | 12.5% |
| 10 GbE | 17 ms | **505 t/s** | 1.6% | 3.4% |
Delta: 1 -> 2.5 GbE is **+18.6% prefill**; 2.5 -> 10 GbE is **+9.4%**.
### 0.4 The KV snapshot: where the link really saturates
On task release, `llama-server` saves the conversation to the prompt
cache. The KV for the 11 FA layers living on the worker GPU must be
copied **worker -> head** for that snapshot, and the slot is BLOCKED
while it runs (`PARALLEL=1` = one user queues behind it).
Measured for a 98,100-token conversation: 3,454 MB copied (model check:
98100 x 11 x 3136 B = 3.38 GB, matches), at line rate (254 MB/s avg,
308 MB/s peak).
| Link | snapshot duration (slot blocked) |
| ---- | -------------------------------- |
| 1 GbE | ~31 s |
| 2.5 GbE | 13.6 s (measured; journal: "prompt cache update took 14418 ms") |
| 10 GbE | ~5 s |
This is the only phase that runs the link at ~100% utilization for a
long time. Prefill runs at 12-26% of line rate in bursts (per ubatch)
and ~6-16% on average; decode runs at ~2% or less.
### 0.5 Decode (MTP verify step, 5 positions)
| Link | net time per step | share of 164 ms step | decode t/s |
| ---- | ----------------- | -------------------- | ---------- |
| 1 GbE | 4.0 ms (RTT 0.74 + wire 3.4) | 2.4% | ~23.0 |
| 2.5 GbE | 1.9 ms (RTT 0.59 + wire 1.4) | 1.1% | 23.47 (measured) |
| 10 GbE | 0.6 ms (RTT ~0.25 + wire 0.3) | 0.4% | ~23.5 |
Decode is latency-touched but bandwidth-trivial. Average wire rate
~19 Mbit/s in each direction.
### 0.6 Verdict for the 10 GbE USB-C adapter question
- The 1 GbE -> 2.5 GbE upgrade (the big win) is already captured for
free: the claw's native 2.5 GbE `eno1` now carries the RPC path
(the 1 GbE USB adapter was unplugged 2026-08-17; routing moved the
path, the RPC TCP re-attached on the new interface).
- 2.5 -> 10 GbE buys +9.4% prefill and 13.6 s -> 5 s snapshot blocking.
It requires the L2 device at 192.168.0.1 to be 10G-capable, or a
direct cable between the two machines (dedicated link, second IP
pair, or a lower-metric static route).
- At current usage (PARALLEL=1, decode-heavy), 2.5 GbE is comfortable.
## 1. Bandwidth Math: Bytes per Token per Layer Boundary
### 1.1 Pipeline Parallelism (PP)
During autoregressive generation, each transformer layer produces a hidden state of shape `(batch_size, hidden_size)`. At a pipeline stage boundary, this tensor must be communicated between nodes.
**Forward pass only (inference):**
```
bytes_per_boundary = batch_size × hidden_size × 2 [fp16 = 2 bytes]
```
**Key detail:** During *generation* (one token at a time), seq_len = 1 at each boundary crossing. The KV cache stays local. The hidden state is `(batch, hidden_size)` — independent of context length.
**For a 27B dense model (e.g., Qwen3.6-27B):**
- Hidden size: H = 4096 (standard for Llama/Qwen-style architectures).
Note: the Qwen3.8-27B actually deployed on this fleet has H = 5120
(GGUF-verified); see section 0, which supersedes these theoretical
numbers with measurements.
- Per-token, per-boundary: `4096 × 2 = 8,192 bytes = 8 KB`
- With Nₚₚ stage boundaries: total = `Nₚₚ × 8 KB/tok`
**For a 36B MoE model (e.g., Qwen3.6-35B-A3B):**
- Hidden size: H = 6144 (common for this scale)
- Layers: L = 48 (typical)
- Per-token, per-boundary: `6144 × 2 = 12,288 bytes = 12 KB`
- With Nₚₚ stage boundaries: total = `Nₚₚ × 12 KB/tok`
**Full example — 27B model, 2 PP stages (1 boundary), batch=1:**
```
1 boundary × 4096 × 2 bytes = 8,192 bytes/tok
```
**Example — same model, batch=64:**
```
1 boundary × 4096 × 64 × 2 bytes = 524,288 bytes = 512 KB/tok
```
**Example — 36B model, 2 PP stages, batch=1:**
```
1 boundary × 6144 × 2 bytes = 12,288 bytes = 12 KB/tok
```
### 1.2 Tensor Parallelism (TP) — for comparison
In tensor parallelism, each MLP and attention layer performs an all-reduce on the hidden state:
```
bytes_per_layer = batch_size × hidden_size × 2 [one all-reduce per layer]
total_bytes_per_tok = L × batch_size × hidden_size × 2 [L all-reduces total]
```
**For 27B, H=4096, L=32, TP=2, batch=1:**
```
32 layers × 4096 × 1 × 2 bytes = 262,144 bytes = 256 KB/tok
(16 KB per layer all-reduce × 32 layers)
```
**For 36B, H=6144, L=48, TP=2, batch=1:**
```
48 layers × 6144 × 1 × 2 bytes = 589,824 bytes = 576 KB/tok
(12 KB per layer all-reduce × 48 layers)
```
### 1.3 Scaling with batch size
| Model | H | L | tok/s | Batch | PP boundary bytes/tok | TP total bytes/tok |
|-------|-----|-----|-------|-------|----------------------|--------------------|
| 27B | 4096 | 32 | 10 | 1 | 8 KB | 256 KB |
| 27B | 4096 | 32 | 50 | 1 | 8 KB | 256 KB |
| 27B | 4096 | 32 | 100 | 1 | 8 KB | 256 KB |
| 27B | 4096 | 32 | 10 | 16 | 128 KB | 4,096 KB |
| 27B | 4096 | 32 | 50 | 16 | 128 KB | 4,096 KB |
| 27B | 4096 | 32 | 100 | 16 | 128 KB | 4,096 KB |
| 27B | 4096 | 32 | 10 | 64 | 512 KB | 16,384 KB |
| 27B | 4096 | 32 | 50 | 64 | 512 KB | 16,384 KB |
| 27B | 4096 | 32 | 100 | 64 | 512 KB | 16,384 KB |
| 36B | 6144 | 48 | 10 | 1 | 12 KB | 576 KB |
| 36B | 6144 | 48 | 50 | 1 | 12 KB | 576 KB |
| 36B | 6144 | 48 | 100 | 1 | 12 KB | 576 KB |
| 36B | 6144 | 48 | 10 | 16 | 192 KB | 9,216 KB |
| 36B | 6144 | 48 | 50 | 16 | 192 KB | 9,216 KB |
| 36B | 6144 | 48 | 100 | 16 | 192 KB | 9,216 KB |
| 36B | 6144 | 48 | 10 | 64 | 768 KB | 36,864 KB |
| 36B | 6144 | 48 | 50 | 64 | 768 KB | 36,864 KB |
| 36B | 6144 | 48 | 100 | 64 | 768 KB | 36,864 KB |
---
## 2. Required Bandwidth vs. Ethernet Capabilities
### 2.1 Ethernet link capacities
| Link | Raw Rate | Usable TCP (approx.) | MB/s usable |
|------|----------|---------------------|-------------|
| 1 GbE | 1,000 Mbit/s | ~940 Mbit/s | ~118 MB/s |
| 2.5 GbE | 2,500 Mbit/s | ~2,350 Mbit/s | ~294 MB/s |
| 10 GbE | 10,000 Mbit/s | ~9,400 Mbit/s | ~1,175 MB/s |
### 2.2 Required bandwidth (Mbps) by model, batch, and throughput
**27B model (H=4096), 1 PP boundary:**
| tok/s | Batch=1 | Batch=16 | Batch=64 |
|-------|---------|----------|----------|
| 10 | 0.13 Mbit/s | 2.05 Mbit/s | 8.19 Mbit/s |
| 50 | 0.65 Mbit/s | 10.24 Mbit/s | 40.96 Mbit/s |
| 100 | 1.31 Mbit/s | 20.48 Mbit/s | 81.92 Mbit/s |
**36B model (H=6144), 1 PP boundary:**
| tok/s | Batch=1 | Batch=16 | Batch=64 |
|-------|---------|----------|----------|
| 10 | 0.20 Mbit/s | 3.07 Mbit/s | 12.29 Mbit/s |
| 50 | 0.98 Mbit/s | 15.36 Mbit/s | 61.44 Mbit/s |
| 100 | 1.97 Mbit/s | 30.72 Mbit/s | 122.88 Mbit/s |
**27B model, TP (L=32 layers all-reduce):**
| tok/s | Batch=1 | Batch=16 | Batch=64 |
|-------|---------|----------|----------|
| 10 | 2.05 Mbit/s | 32.77 Mbit/s | 131.07 Mbit/s |
| 50 | 10.24 Mbit/s | 163.84 Mbit/s | 655.36 Mbit/s |
| 100 | 20.48 Mbit/s | 327.68 Mbit/s | 1,310.72 Mbit/s |
### 2.3 Feasibility matrix
Green = feasible (< 50% of link capacity). Yellow = marginal (50–90%). Red = saturated (> 90%).
**27B model — PP (1 boundary), batch=1:**
| | 1 GbE (118 MB/s) | 2.5 GbE (294 MB/s) | 10 GbE (1175 MB/s) |
|---|---|---|---|
| 10 tok/s | ✅ 0.13 Mbit/s | ✅ 0.13 Mbit/s | ✅ 0.13 Mbit/s |
| 50 tok/s | ✅ 0.65 Mbit/s | ✅ 0.65 Mbit/s | ✅ 0.65 Mbit/s |
| 100 tok/s | ✅ 1.31 Mbit/s | ✅ 1.31 Mbit/s | ✅ 1.31 Mbit/s |
**27B model — PP (1 boundary), batch=16:**
| | 1 GbE | 2.5 GbE | 10 GbE |
|---|---|---|---|
| 10 tok/s | ✅ 2.05 Mbit/s | ✅ 2.05 Mbit/s | ✅ 2.05 Mbit/s |
| 50 tok/s | ✅ 10.24 Mbit/s | ✅ 10.24 Mbit/s | ✅ 10.24 Mbit/s |
| 100 tok/s | ✅ 20.48 Mbit/s | ✅ 20.48 Mbit/s | ✅ 20.48 Mbit/s |
**27B model — PP (1 boundary), batch=64:**
| | 1 GbE | 2.5 GbE | 10 GbE |
|---|---|---|---|
| 10 tok/s | ✅ 8.19 Mbit/s | ✅ 8.19 Mbit/s | ✅ 8.19 Mbit/s |
| 50 tok/s | ✅ 40.96 Mbit/s | ✅ 40.96 Mbit/s | ✅ 40.96 Mbit/s |
| 100 tok/s | ✅ 81.92 Mbit/s | ✅ 81.92 Mbit/s | ✅ 81.92 Mbit/s |
**27B model — TP (32 layers), batch=1:**
| | 1 GbE | 2.5 GbE | 10 GbE |
|---|---|---|---|
| 10 tok/s | ✅ 2.05 Mbit/s | ✅ 2.05 Mbit/s | ✅ 2.05 Mbit/s |
| 50 tok/s | ✅ 10.24 Mbit/s | ✅ 10.24 Mbit/s | ✅ 10.24 Mbit/s |
| 100 tok/s | ✅ 20.48 Mbit/s | ✅ 20.48 Mbit/s | ✅ 20.48 Mbit/s |
**27B model — TP (32 layers), batch=16:**
| | 1 GbE | 2.5 GbE | 10 GbE |
|---|---|---|---|
| 10 tok/s | ✅ 32.77 Mbit/s | ✅ 32.77 Mbit/s | ✅ 32.77 Mbit/s |
| 50 tok/s | ✅ 163.84 Mbit/s | ✅ 163.84 Mbit/s | ✅ 163.84 Mbit/s |
| 100 tok/s | ✅ 327.68 Mbit/s | ✅ 327.68 Mbit/s | ✅ 327.68 Mbit/s |
**27B model — TP (32 layers), batch=64:**
| | 1 GbE | 2.5 GbE | 10 GbE |
|---|---|---|---|
| 10 tok/s | ✅ 131.07 Mbit/s | ✅ 131.07 Mbit/s | ✅ 131.07 Mbit/s |
| 50 tok/s | ✅ 655.36 Mbit/s | ✅ 655.36 Mbit/s | ✅ 655.36 Mbit/s |
| 100 tok/s | ✅ 1310.72 Mbit/s | ✅ 1310.72 Mbit/s | ✅ 1310.72 Mbit/s |
**27B TP batch=64 at 100 tok/s: 1310.72 Mbit/s > 1 GbE usable (1000 Mbit/s) — link-saturated on 1 GbE**
**27B TP batch=64 at 100 tok/s: 1310.72 Mbit/s ≈ 2.5 GbE usable (2500 Mbit/s) — ~52% saturated**
**Key finding:** PP is feasible across all ethernet classes for all batch sizes up to at least 100 tok/s. TP becomes bottlenecked on 1 GbE at batch=64 and 100 tok/s.
---
## 3. Why PP is the Only Realistic Option over Ethernet
### 3.1 Communication cost comparison
| Scheme | Communication per tok per rank | 27B, H=4096, batch=1 | 36B, H=6144, batch=1 |
|--------|------|---|---|
| PP (Nₚₚ boundaries) | Nₚₚ × H × 2 | Nₚₚ × 8 KB | Nₚₚ × 12 KB |
| TP (L layers all-reduce) | L × H × 2 | 32 × 8 KB = 256 KB | 48 × 12 KB = 576 KB |
PP communication per token is **L/Nₚₚ times smaller** than TP communication per token (factor of ~16× for 2 PP stages).
### 3.2 Latency-bound nature of autoregressive generation
During autoregressive generation, each layer computes one token at a time. In TP, every layer requires an all-reduce (2× ring latency or 1× tree latency per layer). For L=32 layers, that's 32 synchronization points per token.
In PP, activations cross the stage boundary only once per layer (or L/Nₚₚ times). The critical path is:
```
Stage 0 computes → send hidden state across boundary → Stage 1 computes → ...
```
This is a single bottleneck per token, with latency proportional to network RTT × number of stages, not L.
### 3.3 Bandwidth utilization
With PP, each boundary crossing transfers `H × 2` bytes. For 8 KB at 1 GbE, the transfer completes in ~68 µs. This is negligible compared to compute time (milliseconds per layer on a GPU). The link is underutilized.
With TP, each layer transfers `H × 2` bytes but this happens L times. For 256 KB total at 1 GbE, that's ~2 ms of pure transfer time — potentially comparable to compute time on a fast GPU.
### 3.4 Real-world evidence
- Petals uses PP exclusively over the public internet and achieves usable inference at 1–2 tok/s even with 100 Mbit/s links [1].
- llama.cpp RPC splits layers across nodes in pipeline order, not tensor sharding [2].
- exo uses TP but achieves speedup only with RDMA over Thunderbolt (~50 Gbit/s) [3].
- Petals paper Table 3: inference performance does **not depend much on bandwidth** (1 Gbit/s vs 100 Mbit/s gives nearly identical results) but **degrades significantly with higher latency** [1].
**Conclusion:** For gigabit-class ethernet, PP is the only practical multi-node parallelism strategy for inference. TP can work on 10 GbE for small batches but saturates at 1 GbE with moderate batch sizes.
---
## 4. Real-World Multi-Node Inference Benchmarks
### 4.1 Summary table
| System | Model | Hardware | # Nodes | Network | tok/s (batch=1) | Notes |
|--------|-------|----------|---------|---------|-----------------|-------|
| Petals (local) | BLOOM-176B | 3× A100 80GB | 3 | 1 Gbit/s, <5 ms RTT | 1.71 | PP, 22 stages, quantized [1] |
| Petals (local) | BLOOM-176B | 3× A100 80GB | 3 | 100 Mbit/s, <5 ms RTT | 1.66 | ~3% slowdown vs 1 Gbit/s [1] |
| Petals (local) | BLOOM-176B | 3× A100 80GB | 3 | 100 Mbit/s, 100 ms RTT | 1.23 | Latency matters more than BW [1] |
| Petals (wan) | BLOOM-176B | 14 consumer GPUs | 14 | Real internet, 100–1000 Mbit/s | 0.83 | Europe+N. America [1] |
| Petals | Llama 2 70B | consumer GPUs | various | Internet | up to 6 | Interactive chat [4] |
| Petals | Falcon 180B | consumer GPUs | various | Internet | up to 4 | Interactive chat [4] |
| llama.cpp RPC | Qwen3-235B | 4× M3 Ultra Mac Studio | 4 | Ethernet (2.5 GbE) | ~20 | Scales down with more nodes [3] |
| llama.cpp RPC | DeepSeek v3.1 | 4× M3 Ultra Mac Studio | 4 | Thunderbolt 5 | ~25 | Scales down with more nodes [3] |
| llama.cpp RPC | Kimi K2 (1T) | 4× M3 Ultra Mac Studio | 4 | Thunderbolt 5 | ~30 | A32B sparse, 32B active [3] |
| Exo (RDMA) | Qwen3-235B | 4× M3 Ultra Mac Studio | 4 | TB5 RDMA (~50 GbE) | 32 | TP, scales UP with more nodes [3] |
| Exo (RDMA) | DeepSeek v3.1 | 4× M3 Ultra Mac Studio | 4 | TB5 RDMA (~50 GbE) | ~30 | 671B dense TP [3] |
| vLLM PP | Llama-3-70B | 2× A100 80GB | 2 | NVLink + Ethernet | ~15 | PP across nodes, TP within [5] |
### 4.2 Detailed notes
**Petals [1]:**
- Uses pipeline parallelism with dynamic 8-bit quantization (halves bandwidth)
- For BLOOM-176B (H=14336), per-boundary communication is ~14 KB/tok with quantization
- Recommended bandwidth: 25 Mbit/s bidirectional for interactive inference [1]
- Key finding: "For inference, performance does not depend much on bandwidth or sequence length but degrades with higher latency" — RTT matters more than throughput
- At 1 Gbit/s with <5 ms RTT: 1.71 steps/s; at 100 Mbit/s with <5 ms: 1.66 steps/s (only 3% difference)
- At 100 Mbit/s with 100 ms RTT: 1.23 steps/s (28% degradation)
**llama.cpp RPC [2]:**
- Model weights and KV cache distributed across nodes proportionally to memory
- Default: pipeline-style distribution of layers (first layers on node A, last on node B)
- Each node runs `ggml-rpc-server`, main host uses `--rpc` flag
- RDMA transport supported (RoCEv2), negotiated automatically with libibverbs
- Over ethernet, performance decreases with more nodes (each extra node adds a hop)
- Over RDMA/TB5, performance scales up with more nodes (less latency per hop)
**Exo [3]:**
- Uses tensor parallelism with MLX distributed backend
- With RDMA over Thunderbolt 5: up to 1.8× speedup on 2 devices, 3.2× on 4 devices
- Topology-aware auto-parallelism considers latency/bandwidth between links
- Thunderbolt 5 provides ~50–60 GbE real-world throughput (RDMA)
**Key insight from all systems:** Ethernet (1 Gbit/s) is sufficient for PP inference of 20–40B models because per-token activation traffic is small (8–12 KB/boundary). The bottleneck is **latency** (RTT between nodes), not bandwidth.
---
## 5. Overheads
### 5.1 TCP/IP overhead
| Overhead | Amount |
|----------|--------|
| Ethernet header + trailer | 18 bytes (14 + 4 FCS) |
| IPv4 header (no options) | 20 bytes |
| TCP header (no options) | 20 bytes |
| Total per packet | ~58 bytes overhead |
| MTU (jumbo frames 9000) | 9000 bytes payload → 0.6% overhead |
| MTU (1500 standard) | 1500 bytes payload → 3.9% overhead |
**Practical impact:** For 8 KB transfers (27B PP boundary), use jumbo frames where possible to minimize TCP overhead. For small transfers (< 1.5 KB), a single TCP packet is used and the overhead is 58/8192 = 0.7% per transfer — negligible.
### 5.2 RPC/serialization overhead
- llama.cpp RPC: Protobuf serialization adds ~10–20% overhead for small tensors
- Petals: PyTorch tensor serialization over hivemind/gRPC, adds serialization time
- vLLM PP: gRPC with protobuf, similar overhead
- Serialization scales with tensor size: negligible for 8 KB+ tensors
### 5.3 Pipeline bubble overhead
With Nₚₚ pipeline stages and microbatching, the pipeline bubble fraction is:
```
bubble_fraction = (Nₚₚ - 1) / (Nₚₚ + microbatches - 1)
```
For 2 stages, 1 microbatch (single tok): bubble = 1/2 = 50% idle time.
For 2 stages, 8 microbatches: bubble = 1/9 = 11%.
In autoregressive generation with batch=1 and no microbatching, each stage processes one token sequentially:
- Stage 0 finishes, sends hidden state, waits for Stage 1
- Stage 1 processes, returns result, Stage 0 processes next token
- Effective utilization ≈ 1/Nₚₚ if no pipelining overlap
**Mitigation:** Process multiple tokens before waiting. Send next token to Stage 0 while Stage 1 is computing — pipelining reduces bubble.
### 5.4 Per-hop LAN latency
| Network | Typical RTT | One-way latency |
|---------|------------|-----------------|
| 1 GbE (Cat6) | 30–50 µs | 15–25 µs |
| 2.5 GbE (Cat6) | 30–50 µs | 15–25 µs |
| 10 GbE (Cat6a) | 20–30 µs | 10–15 µs |
| RoCEv2 (IB/25GbE) | 1–5 µs | <5 µs |
| RDMA (Thunderbolt 5) | 10–30 µs round-trip end-to-end | — |
**Impact on PP inference:** With Nₚₚ = 2 stages and batch=1, one token takes:
```
t_compute + t_roundtrip + t_compute
```
At 30 µs RTT and ~2 ms layer compute time per GPU: the network adds ~30 µs, which is 1.5% of total. Negligible at LAN speeds.
With Nₚₚ = 4 and 30 µs per-hop RTT: ~120 µs network, still <6% of compute time.
### 5.5 Quantization overhead
- Petals uses dynamic blockwise 8-bit quantization: halves hidden state bandwidth from 16-bit to 8-bit [1]
- For 27B: 8 KB → 4 KB per boundary with 8-bit quantization
- Accuracy impact: <1% on standard benchmarks [1]
- Quantization/dequantization adds ~1–5 ms per stage on modern GPUs (negligible vs. compute)
---
## 6. Conclusions and Recommendations
### For the 27B / 36B models on gigabit-class ethernet:
1. **1 GbE is sufficient for PP inference** up to at least 100 tok/s at batch=64 (81.92 Mbit/s required, well under 1000 Mbit/s capacity). Even with TCP/IP and RPC overheads, margin is >90%.
2. **Pipeline parallelism is the correct choice** over ethernet — PP sends 8–12 KB per boundary vs. 256–576 KB for TP, a 32× advantage for 2 PP stages.
3. **Latency matters more than bandwidth** — the Petals paper demonstrates that dropping from 1 Gbit/s to 100 Mbit/s only reduces throughput by ~3%, while increasing RTT from 5 ms to 100 ms reduces it by ~28%.
4. **With 8-bit quantization**, bandwidth requirements halve, providing additional safety margin.
5. **Jumbo frames recommended** to minimize TCP overhead on larger transfers.
6. **For batch sizes ≥ 16**, 1 GbE remains viable for PP but consider 2.5 GbE for headroom and faster model loading.
---
## References
[1] Borzunov et al., "Petals: Collaborative Inference and Fine-tuning of Large Models," ACL 2023. arXiv:2209.01188. Available at: https://arxiv.org/pdf/2209.01188 — Primary benchmark source: Table 3 shows inference throughput at various bandwidths and latencies for BLOOM-176B. Key finding: inference insensitive to bandwidth, sensitive to latency. Recommended 25 Mbit/s bidirectional bandwidth.
[2] ggml-org/llama.cpp RPC documentation. Available at: https://github.com/ggml-org/llama.cpp/blob/master/tools/rpc/README.md — Architecture for distributed LLM inference via RPC. Supports RDMA transport (RoCEv2), automatic negotiation, local tensor caching.
[3] exo-explore/exo README. Available at: https://github.com/exo-explore/exo — Multi-node LLM inference on Apple Silicon with TP and RDMA over Thunderbolt 5. Benchmarks with Qwen3-235B, DeepSeek v3.1, Kimi K2 on 4× M3 Ultra Mac Studio.
[4] Petals chatbot demo and health monitor. Available at: https://chat.petals.dev, https://health.petals.dev — Real-world performance of Llama 2 70B (up to 6 tok/s) and Falcon 180B (up to 4 tok/s) on public swarm.
[5] vLLM pipeline parallel documentation. Available at: https://docs.vllm.ai/en/stable/dev/engine/llm_engine.html — vLLM supports PP across nodes using gRPC with CUDA-graph support for low-latency stage transitions.
---
## Appendix: Full bandwidth formulae
### Pipeline Parallelism (generation, batch=1)
```
bytes_per_boundary_per_tok = hidden_size × 2 [fp16]
total_bytes_per_tok = num_boundaries × hidden_size × 2
bandwidth_mbps = total_bytes_per_tok × tokens_per_second × 8 / 1,000,000
```
### Pipeline Parallelism (generation, batch=N)
```
bytes_per_boundary_per_tok = hidden_size × N × 2
total_bytes_per_tok = num_boundaries × hidden_size × N × 2
```
### Tensor Parallelism (generation, batch=1)
```
bytes_per_layer_allreduce = hidden_size × 2
total_bytes_per_tok = num_layers × hidden_size × 2
bandwidth_mbps = total_bytes_per_tok × tokens_per_second × 8 / 1,000,000
```
### Tensor Parallelism (generation, batch=N)
```
total_bytes_per_tok = num_layers × hidden_size × N × 2
```
### With 8-bit quantization (like Petals)
```
# Simply divide all fp16 byte counts by 2
bytes_per_boundary_per_tok = hidden_size × 1 [int8]
```
serverAppConfigs-docs/LLM/Pipeline Parallelism/Expected Performance.md
-97
View File
@@ -1,97 +0,0 @@
---
type: reference
title: Expected Performance
description: Estimated tokens per second for a 25-40B model split across two nodes over 1/2.5/10 GbE, with real benchmark anchors.
tags:
- pipeline-parallelism
- performance
- estimate
timestamp: 2026-08-14
---
# Expected Performance
This page estimates the speed of a 25-40B model split across two machines
over ethernet. It uses the bandwidth math in [[Ethernet Bandwidth Math]]
and the measured numbers in [[Mixed-Vendor Experiments]].
## The Bandwidth Ceiling Is Not the Limit
A pipeline boundary moves only the hidden state. For a 27B model (hidden
5120), that is 10240 bytes per token per boundary in fp16. Even at 64
tokens per second with batch 64, the traffic is about 262 Mbit/s. 1 GbE can
carry it. The real limits are latency, RPC overhead, and the slowest node.
The Petals paper measured this directly. Dropping from 1 Gbit/s to
100 Mbit/s changed throughput by 3 percent. Raising round-trip latency from
5 ms to 100 ms dropped throughput by 28 percent. Latency matters more than
bandwidth for pipeline inference.
## Real Benchmark Anchors
| Experiment | Model | Network | Decode (t/s) |
| ---------- | ----- | ------- | ------------ |
| Mac Studio + DGX Spark (Metal+CUDA) | Qwen2.5-72B Q4 | 10 GbE | 6 |
| Mac Studio + DGX Spark (Metal+CUDA) | Qwen2.5-7B Q4 | 10 GbE | 53 |
| RTX 4070 + MI50 + 3050 (CUDA+ROCm) | Qwen3-VL-30B Q5 | 1 GbE | 51.5 clean, 17.8 at 32k ctx |
| Two Strix Halos (ROCm RPC) | MiniMax-M2.5-139B Q8 | USB4 | 15.35 |
| AMD four-node Strix Halo (5 GbE) | Kimi K2.5 (1T) | 5 GbE | 8.30 |
| Tesla V100 + R9700x2 (Vulkan, one box) | Qwen3.6-35B-A3B Q8 | PCIe | 62 |
Pattern: RPC decode runs about 2-5 times slower than the best single node.
Prefill benefits from more compute across nodes. The 30B-class model over
1 GbE with warm cache reached 51.5 t/s on a short context and 17.8 t/s at
32k context.
## Estimate for Qwen3.6-27B Q8, Two Nodes
Scenario: RTX 3090 (24 GB VRAM, NVIDIA desktop) plus Strix Halo NUC
(Radeon 8060S, 30 GB physical RAM). The model is the dense 27B at Q8
(29-32 GB). Each node holds part of the weights and KV cache.
| Link | Decode (t/s) | Prefill (t/s) | Notes |
| ---- | ------------ | ------------- | ----- |
| 1 GbE | 8-15 | 200-500 | Latency-bound; per-token RPC round trip |
| 2.5 GbE | 10-18 | 250-600 | Some headroom; latency still dominant |
| 10 GbE | 12-20 | 300-800 | Closest to the published 10 GbE runs |
The dense 27B decodes slower per token than the MoE 30B A3B class. Expect
the lower half of the ranges at full context.
### With Speculative Decoding
Speculative decoding (MTP, DFlash, or ngram-mod) raises effective decode.
Tightwad measured 1.86x on a slow RPC pool. Estimate the effective rate at
1.4-1.8x the raw RPC decode:
| Link | Raw decode (t/s) | With spec (t/s) |
| ---- | ---------------- | --------------- |
| 1 GbE | 8-15 | 12-24 |
| 2.5 GbE | 10-18 | 15-28 |
| 10 GbE | 12-20 | 18-32 |
## Context That Fits
The two real machines give about 51 GiB of usable memory combined
(24 GiB VRAM plus about 27 GiB on the Strix Halo, which has 30 GiB physical
RAM; the 96 GiB driver ceiling is not physical). A 29-32 GB model leaves
about 20-22 GiB for KV cache. The Qwen3.6-27B KV cache costs about 85-100
KiB per token with q8_0 K and f16 V (measured on the large service at
602100 tokens = about 50 GB). That budget gives a total context pool of
roughly 200k-260k tokens across both nodes.
## Caveats
- Every node must run the exact same llama.cpp build. Version skew hangs
the handshake.
- The slowest node sets the pace for every token.
- Model load crosses the network once per start. Expect minutes, not
seconds, for the split load.
- These are estimates. The exact combination of an RTX 3090 desktop and a
Strix Halo NUC has no published benchmark yet.
## See Also
- [[Ethernet Bandwidth Math]]
- [[Mixed-Vendor Experiments]]
- [[Sample Script Set]]
serverAppConfigs-docs/LLM/Pipeline Parallelism/Mixed-Vendor Experiments.md
-9
View File
@@ -1,9 +0,0 @@
---
type: reference
title: "Mixed-Vendor Experiments"
description: "Real-world NVIDIA + AMD Strix Halo multi-node experiments: what works (llama.cpp RPC, Vulkan split), what breaks (mixed CUDA+ROCm vLLM), measured tok/s."
tags:
- pipeline-parallelism
- research
timestamp: 2026-08-14
---
serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.6 27B Spec Methods.md
-9
View File
@@ -1,9 +0,0 @@
---
type: reference
title: "Qwen3.6 27B Spec Methods"
description: "Qwen3.6-27B Q8 GGUF variants and llama.cpp speculative decoding (draft-mtp, draft-dflash, ngram-mod): flags, RPC compatibility, memory/context math."
tags:
- pipeline-parallelism
- research
timestamp: 2026-08-14
---
serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.8 PP Deployment.md
-313
View File
@@ -1,313 +0,0 @@
---
type: reference
description: "Split Qwen3.8-27B across the Strix Halo iGPU (head) and the RTX"
tags:
- llm
- pipeline
timestamp: 2026-09-03
---
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
> See [frank-redo-2026-08-17.md](../frank-redo-2026-08-17.md) and
> [deployment.md](../deployment.md) for the active configuration.
>
> **2026-10-18: nothing pipeline-parallel runs on the fleet.** No RPC listener
> exists on either host (`ss -ltn` shows no :50052), every artifact named below
> is gone (§Files), and the worker hardware left the box: the RTX 3090 was
> removed from Dankserver on 2026-09-15 (it now sits in Dankclaw) and the server
> runs 2× R9700. The live multi-GPU scheme on Dankserver is tensor parallelism
> inside one process ([[Dual-R9700-Cooperation]],
> [[frank-archon]]).
# Qwen3.8-27B Pipeline-Parallel Deployment (Dankclaw + Dankserver)
Split Qwen3.8-27B across the Strix Halo iGPU (head) and the RTX
3090 (worker) using llama.cpp RPC. This page is the design record and the
morning-of runbook. Research background:
[Qwen3.8 Strix optimizations](../qwen3.8-strix-optimizations.md).
## Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68)
This split is the production `frank-dragoon` service. Current
state, verified against the live journal and output smoke tests
(2026-08-17, load 02:06):
- Model: `Qwen3.8-27B-Uncensored-Q6_K.gguf` (22.43 GB, imatrix, no
bundled MTP head). Draft: `Qwen3.8-27B-Uncensored-draft-Q8_0.gguf`
(3.16 GB) via `--spec-draft-model`, running on ROCm0 only (1718 MiB;
zero worker impact — measured, not assumed).
- Split `0.68,0.32` = 45 of 65 blocks (0-44) on the 3090 — 69% of the
layer weight. Maximum for this config; see "Split limits" below.
- KV: K `q8_0` + V `f16` (mismatched), 784 MiB per FA layer: RPC0 8624
MiB (11 FA) + ROCm0 3920 MiB (5 FA). Flash Attention ENABLED despite
the mismatch (`resolve_fused_ops: Flash Attention enabled`) via the
GGML_CUDA_FA_ALL_QUANTS rebuild — see "FA-all-quants rebuild" below.
- Pool `262144`, `PARALLEL=1`, live `n_ctx_slot = 262144`.
* MTP: standalone draft head, acceptance 0.83 then 0.75, mean len 4.0-4.3,
per-position ~0.86/0.71/0.71/0.71 — stronger at every position than the old bundled
head, which measured 0.739/0.554/0.427/0.340.
- Spec chain as deployed: `--spec-draft-n-max 4` on the standalone draft, with
the `--spec-default` ngram-mod fallback at n_match 24 / n_min 48 / n_max 64
(the preset that survived the DSpark/DFlash retirements —
[[qwen3.8-spec-matrix]]).
- Batch 8192 / ubatch 256, `--direct-io`, `--fit off`, `--gpu-layers all`. The
pi `models.json` declared 262144 (`contextWindow = maxTokens`, the house
convention of that era).
- Head GPU side: **ROCm0** (no `--device` flag; HIP registers before
Vulkan).
- Worker steady state: 23854/24576 MiB used, ~273 MiB free (tight but
stable; all allocations are upfront).
- Unit `frank-dragoon.service`, launched through
`serve-pp-head.sh` + `frank-dragoon.env`.
## Service enablement (2026-08-17)
The PP pair is the only inference service enabled for boot. Both units
(`frank-dragoon` on Dankclaw, `frank-pp-worker` on Dankserver) are
enabled. The NPU unit (`frank-zealot` on Dankclaw) and all retired
units are disabled. Restart order: worker first, then head. The Dankserver
helper `frank-restart-small.sh` is obsolete (it restarts the removed
`frank-zealot` unit); restart the worker with
`sudo systemctl restart frank-pp-worker`. The Dankclaw helper
`frank-restart-large.sh` (sync + cache drop + restart) is correct for the
head. The "deploy.sh dankserver is broken until retargeted" note is long fixed:
`./scripts/deploy.sh dankserver --service frank-archon` is the documented
invocation today (`scripts/deploy.sh:38`, dispatch case
`dankserver:frank-archon` at `scripts/deploy.sh:149`). See
[[deployment]] for the full boot-policy table.
## FA-all-quants rebuild (2026-08-17)
Flash Attention with mismatched K/V cache types requires the
`GGML_CUDA_FA_ALL_QUANTS` compile definition. Without it,
`ggml/src/ggml-cuda/fattn.cu` gates FA off when `K->type != V->type`
(`#ifndef GGML_CUDA_FA_ALL_QUANTS`). With it, the K==V check is skipped
and all quantized-K combos are compiled. The head (HIP gfx1151) and the
worker (CUDA sm_86) compile the same sources, so the same CMake flag
covers both:
cmake -S . -B build -DGGML_CUDA_FA_ALL_QUANTS=ON
cmake --build build -j --target llama-server # dankclaw
cmake --build build -j --target ggml-rpc-server # dankserver
Both hosts at commit 9b05354ec (b10433). The flag was already cached
ON in the worker build (Aug 16 objects carry it); the head required a
full ggml-hip rebuild. Keep both binaries on the same commit — version
skew hangs the RPC handshake.
## Split limits (measured 2026-08-17, Q6_K + q8_0/f16 KV)
KV per FA layer at 262144 = 784 MiB (K q8_0 1088 B/token + V f16 2048
B/token). Layer pool 20334 MB (GGUF scan, exact), embd on the head side
(RPC0 buffer 10429.88 MiB at f=0.53 -> the split pool excludes embd).
- f=0.53 -> 35 blocks, 8 FA, KV 6272 MiB, 5721 MiB free. Phase 1.
- f=0.68 -> 45 blocks (0-44), 11 FA (3..43), KV 8624 MiB, 273 MiB
free. PRODUCTION (the max): 46 blocks leaves ~0.3 GiB; 48 blocks
(f >= 0.7197, the round(f x 66) boundary) adds FA block 47 -> KV
9408 -> OOM by ~0.85 GiB.
The KV-per-device count = round(f x 66) layers, independent of the
weight boundary (same rule as the Q8_K_XL config). FA blocks at
3,7,...,63.
## Split limits (measured 2026-08-17, Q8_K_XL + f16/f16 KV) — superseded
The 0.40 split (2026-08-16, 2x200700 pool) was KV-bound at 26 blocks.
With the pool shrunk to 262144, the KV per full-attention layer drops to
exactly 1024 MiB, freeing headroom. Measurements:
- 0.50,0.50 -> 32 blocks (0-31), 8 FA on RPC0, KV 8192 MiB, 1477 MiB
free. Healthy (2026-08-17 00:29 load).
- 0.55,0.45 -> 36 blocks (0-35): the 9th FA block, KV 9216 MiB ->
cudaMalloc OOM by ~740 MiB.
- 0.54,0.46 -> weights 35 blocks (0-34) but KV still 9216 MiB -> OOM.
The KV-per-device count rounds the split fraction over 66 layers
(round(f x 66)), independent of the weight boundary; the 9-FA flip
sits at f ~= 0.538, just below the weight threshold for block 35, so
the 35-block config is unreachable.
- 0.53,0.47 -> 34 blocks (0-33), 8 FA, KV 8192 MiB. Safe under every
rounding rule (round/floor/ceil of 0.53 x 66 = 34.98). Production.
GGUF block scan (exact sizes): FA blocks at 3,7,...,63 (409.6 MB each),
linear blocks 431.3 MB; total layer weight 27720 MB. The split pool
includes the MTP head for weights and token_embd for KV — the two
boundaries diverge, which is why 0.54 split them apart.
## Cache incident (2026-08-17): garbage output from a full worker disk
Symptom: after a split change, the service loaded "healthy" but every
completion degenerated to a repeated junk token (e.g. 48 x "/", empty
content, degenerate logits). Root cause chain:
1. The worker RPC tensor cache keys per session. Each model load writes
a fresh full copy (~15 GB); over ~5 sessions of two models it grew to
4434 files / 96 GB.
2. The Dankserver root disk filled to 100%. Cache writes started
failing, leaving 0-byte entries. A subsequent reload hit corrupt
cache entries and the worker served garbage weights for the
newly-needed layers without validation.
3. Fix: move the cache to .AGENTTRASH (then empty it), restart the
worker, free the disk, reload the head (full re-stream ~2.5 min).
A clean cache and a healthy disk produced correct output immediately
(acceptance 0.89 on the first smoke test).
Guard was `scripts/prune-rpc-cache.sh` on Dankserver (cron, 40 GiB cap). That
script is gone from the repo and from the host, yet the crontab line still
runs every 10 minutes against it
(`*/10 * * * * /home/dank/vLLM/scripts/prune-rpc-cache.sh`, `crontab -l`, read
2026-10-18 — the target does not exist, so it fails silently). Never let the
cache directory grow unchecked; a full disk silently corrupts it. Signature to
recognize: healthy load + degenerate repeated-token output.
## Topology
```
Dankclaw (head, 192.168.0.143) Dankserver (worker, 192.168.0.235)
llama-server b10433 ggml-rpc-server b10433
--rpc 192.168.0.235:50052 --host 192.168.0.235 --port 50052
model file (31.5 GB) -d CUDA0 -c (tensor cache)
[RPC0, ROCm0|Vulkan0] layers RTX 3090: layers 0-44 (45 of 65 blocks)
Strix: layers 45-64 + lm_head
hidden state 10 KiB/position/direction (H=5120, GGUF-verified);
measured wire bytes ~40 KB/position/direction (serialized graph
dominates); 20.5 MB per ubatch both directions. Details:
[[Ethernet Bandwidth Math]] section 0.
```
- **Layer-sequential split** (true pipeline parallelism): layers 0..N run on
the worker, the rest on the head, in order. One hidden-state handoff per
token step. DECODE is RTT-bound, not bandwidth-bound (network is ~1-2.5%
of step time). PREFILL and the prompt-cache KV snapshot ARE
bandwidth-bound: the link runs at line rate for the wire burst of every
ubatch and ~100% during the snapshot copy. Measured numbers:
[[Ethernet Bandwidth Math]] section 0.
- **Why the head is on Dankclaw:** the model file is already there; the
worker streams its tensor share at load (no 31.5 GB copy to the server).
- **Both sides must run the identical build** (b10433 / 9b05354ec). The
worker build lived in a worktree `/home/dank/llama-cpp-rpc` on Dankserver so the
production b10121 build stayed untouched (that path no longer exists, read
2026-10-18). A version skew hangs the RPC handshake with no error.
## Files
| File | Machine | Purpose |
|------|---------|---------|
| `scripts/frank-pp-worker.env` | server | worker tunables (bind, port, device, cache) |
| `scripts/serve-pp-worker.sh` | server | runs `ggml-rpc-server` from the b10433 worktree build |
| `scripts/frank-dragoon.env` | claw | head tunables (split, ctx, spec, KV, I/O); was `frank-pp-head.env` before the 2026-08-16 rename |
| `scripts/serve-pp-head.sh` | claw | runs `llama-server --rpc ...` (see "Gotchas") |
| `scripts/frank-dragoon.service` / `frank-pp-worker.service` | both | optional systemd units (manual install); the head unit was renamed from `frank-pp-head.service` on 2026-08-16 and is symlinked from the repo into `/etc/systemd/system` |
| `scripts/pp-test.sh` | any box with ssh to both | test orchestrator (preflight/start/verify/bench/stop-pp/restore) |
| `scripts/pp-bench.py` | claw | API driver; reports server-side `print_timing` |
**Every PP file in that table except `scripts/frank-dragoon.env` no longer
exists** (checkout read 2026-10-18: `frank-pp-worker.env`,
`serve-pp-worker.sh`, `serve-pp-head.sh`, `frank-pp-worker.service`,
`pp-test.sh`, `pp-bench.py` and `prune-rpc-cache.sh` are all absent), and
`scripts/frank-dragoon.env` is today the halogen-flash-server env, not a PP
head env ([[frank-dragoon]]).
## Gotchas (all verified in b10433 source / live tests, 2026-08-15)
1. **No `--device` on the head.** Setting `params.devices` makes
`llama_prepare_model_devices` use exactly that list; the RPC device is
silently dropped and `--rpc` becomes a no-op. Omit it; default selection
puts RPC first.
2. **The iGPU side is ROCm by default.** Backends register in fixed order
(HIP before Vulkan); the iGPU dedup rule keeps the first backend's view.
Live check showed `[RPC0, ROCm0]`. For a Vulkan side, use the second
head build `/home/dank/llama.cpp/build-vulkan-rpc` (HIP off; removed — the
path is gone on 2026-10-18) — set `HEAD_BIN` in the env. Trade: ROCm side =
faster prefill, ~3% slower decode (measured on this box).
3. **Pin `--tensor-split 1,1`.** The default split is proportional to
*reported free memory*: the Strix reports up to 96 GiB (GTT) vs the
3090's 24 GB, so the default lands ~20/80 on the slow device. 1,1 = 50/50 =
balanced per-layer latency. The 2026-08-15 4B smoke test that proved it
(production untouched): at a 0.2/0.8 split only 4 of 36 layers reached the
3090, and at a real split worker VRAM rose ~819 MiB with 95 t/s prefill /
58.8 t/s decode on the tiny model; link RTT ~0.3–0.5 ms.
4. **`-dio` on the head.** Large model + RPC hangs at load without direct
I/O.
5. **RPC is unauthenticated and unencrypted.** LAN only. The worker binds
0.0.0.0-equivalent by design (`--host 192.168.0.235`).
6. **Env var names.** Never `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`
(HIP runtime reads them; the 2026-08-13 `TEMP` incident).
7. **Page-cache state confounds loads.** `pp-test.sh start` drops caches on
the head before load (warm-cache loads crash fast; cold loads survive).
## Runbook (morning of 2026-08-16)
Run from any box with ssh access to both (repo path on claw:
`/home/dank/vllmScripts`, on server: `/home/dank/vLLM` — both still exist as
stale checkouts; the deployment checkout today is `/home/dank/serverAppConfigs`
on both hosts).
```
# 0. one-time: pull the kit on both boxes
ssh dank@dankclaw 'cd /home/dank/vllmScripts && git pull'
ssh dank@dankserver 'cd /home/dank/vLLM && git pull'
# 1. checks (read-only)
./scripts/pp-test.sh preflight
# 2. full run: stop prod -> start PP -> verify split -> bench -> restore
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh all
# or step by step:
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh start
./scripts/pp-test.sh verify # layer split + worker VRAM
./scripts/pp-test.sh bench 4 256 2 # ~4k prompt, 256 out, 2 reps
./scripts/pp-test.sh bench 16 256 1
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh stop-pp
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh restore
```
Config variants: edit `scripts/frank-dragoon.env` on the claw (git commit +
pull if the bench runs from the repo), then re-run `start` + `bench`:
- No spec baseline: `SPEC_NONE=1`
- Vulkan head side: `HEAD_BIN=/home/dank/llama.cpp/build-vulkan-rpc/bin/llama-server`
- Q4_0 MTP draft head: add `--spec-draft-model /home/dank/models/Qwen3.8-27B-MTP-Q4_0.gguf` to the serve script's SPEC block (env hook: see `SPEC_DRAFT_MODEL` below)
- n_max sweep: `SPEC_DRAFT_N_MAX=3|4|5`
- Plain Q8_0: `MODEL_FILE=Qwen3.8-27B-Q8_0.gguf` (file must be on the claw)
Verification checklist after each `start`:
1. `verify` shows `using device RPC0 ... using device ROCm0|Vulkan0`.
2. Layer assignment lines (`-lv 10`): worker gets the first ~32 layers at a
1,1 split.
3. `nvidia-smi` on the server showed ~15–16 GB used by the worker
(production small service + worker share). No NVIDIA card is in that host
today — the equivalent check is `rocm-smi --showmeminfo vram`.
4. Head GTT usage (rocm-smi) well under 96 GiB; model fully on devices
(`--gpu-layers all` offloads everything; CPU holds embeddings/lm_head glue).
## Expected numbers
Per-layer: 3090 ≈ 2–3× faster than the 8060S. 50/50 split → each token
step ≈ max(32 layers on 3090, 32 layers on Strix) + ~0.5 ms RTT.
- Production reference: 25 t/s @16k (Vulkan, spec, single node).
- Target: **≥ 30 t/s @4k, ≥ 20 t/s @16k** with spec.
- Decode floor: with ngram/mod acceptance, spec multiplies the raw rate;
raw (no spec) rate should clear ~15–18 t/s @16k for the target to hold.
## Rollback
`pp-test.sh restore` starts both production services. If a PP test wedges a
box: `systemctl restart <unit>` after killing the stray processes
(`pkill -x llama-server` / `pkill -x ggml-rpc-server` — never `pkill -f`
with a pattern matching your own ssh command line).
## Open questions for the test day
1. Does the MTP head (which sits in the last layer group, iGPU side) draft
cleanly with the verify pass crossing the RPC boundary? Watch acceptance
in the head log (`draft acceptance = X`).
2. Two parallel slots on the split (PARALLEL=2, CTX 65536): RPC is
serialized per compute — expect the 2-slot speedup to be weaker than
single-node. Measure, do not assume.
3. Prefill parity: ROCm-side head should prefill ≥ single-node ROCm; if not,
the ubatch crossing the RPC boundary is the suspect.
serverAppConfigs-docs/LLM/Pipeline Parallelism/Sample Script Set.md
-261
View File
@@ -1,261 +0,0 @@
---
type: reference
title: Sample Script Set
description: An educational two-node llama.cpp RPC split of Qwen3.6-27B-Q8_K_P over ethernet. For reading only. Do not run or deploy.
tags:
- pipeline-parallelism
- sample
- llamacpp
- educational
timestamp: 2026-08-14
---
# Sample Script Set — Two-Node Qwen3.6-27B Q8 Split
> WARNING. This page is for educational reading only. Do not run these
> scripts. Do not deploy them. They are a template, not a tested product.
> The RPC backend is a proof of concept that the upstream README calls
> "fragile and insecure". Running it on a production network is unsafe.
## Goal
Run a Qwen3.6-27B Q8 model (about 29 to 32 GB of weights) across two
machines using llama.cpp RPC pipeline parallelism. Node A is an NVIDIA
desktop with an RTX 3090. Node B is a Strix Halo NUC (Ryzen AI MAX+ 395,
Radeon 8060S). Use the largest context that fits on both nodes together.
## How the Split Works
```mermaid
graph LR
subgraph NodeA[NVIDIA Desktop — HEAD]
A1[llama-server<br/>--model 27B-Q8.gguf<br/>--rpc nodeB:50052]
end
subgraph NodeB[Strix Halo NUC — WORKER]
B1[ggml-rpc-server<br/>--host nodeB --port 50052<br/>--device CUDA0? no — ROCm]
end
A1 <-->|hidden states over ethernet| B1
```
Node A runs the head `llama-server`. Node B runs `ggml-rpc-server`, which
exposes its GPU as a remote device. The head splits weights and KV cache
across its own GPU and the remote device in proportion to free memory.
Each token generation passes the hidden state once across the boundary.
Key facts that shape the scripts:
- Both nodes must run the exact same llama.cpp build tag. Version skew
hangs the handshake.
- The head build needs its backend plus RPC. The worker build needs its
backend plus RPC. CUDA and ROCm do not mix inside one binary; each node
uses its own backend.
- Speculative decoding runs on the head. The draft model loads on the head.
- The model file must be present on every node with the same bytes.
## File Layout
```
pipeline-pp/
README.md
build.sh # build llama.cpp with RPC on each node
workers/worker-strix.sh # run ggml-rpc-server on the Strix Halo NUC
head/head-nvidia.sh # run llama-server on the NVIDIA desktop
head/frank-27b-pp.env # tunables for the head
head/verify.sh # health and split check
```
## Build Script
File: `build.sh`
```bash
#!/usr/bin/env bash
# Build llama.cpp with RPC support. Run ONE build per node, with that
# node's own backend enabled. Educational only. Do not run in production.
set -euo pipefail
LLAMA_DIR="$1" # path to a llama.cpp checkout
NODE="$2" # "nvidia" or "strix"
cmake -S "$LLAMA_DIR" -B "$LLAMA_DIR/build-pp" \
-DCMAKE_BUILD_TYPE=Release \
-DGGML_RPC=ON \
$( [ "$NODE" = nvidia ] && echo -DGGML_CUDA=ON \
|| echo -DGGML_HIP=ON -DGGML_HIP_PLATFORM=amd ) \
-DGGML_CUDA_NO_GGVK=ON 2>/dev/null || true
cmake --build "$LLAMA_DIR/build-pp" --config Release -j
# Expected binaries: build-pp/bin/ggml-rpc-server, llama-server, llama-cli
```
Notes: the HEAD does not need a GPU if you offload everything to RPC
workers. Keep the head on the NVIDIA desktop so it can also use its local
3090 for part of the split.
## Worker Script (Node B, Strix Halo NUC)
File: `workers/worker-strix.sh`
```bash
#!/usr/bin/env bash
# Expose the Strix Halo GPU as an RPC device for a remote head.
# Educational only. Do not run on an unprotected network.
set -euo pipefail
BIN=/home/dank/llama.cpp/build-pp/bin
# Bind to the LAN address. The default binds 127.0.0.1 only.
HOST=192.168.0.143 # the Strix Halo NUC address
PORT=50052
# -c enables a local model-file cache so the split load runs once, not per boot.
exec "$BIN/ggml-rpc-server" --host "$HOST" --port "$PORT" --cache
# Use --device to name the backend device explicitly if auto-detect fails:
# ggml-rpc-server --host $HOST --port $PORT --device gfx1151
```
Security note: `ggml-rpc-server` exposes all memory to any client that can
reach the port. Keep it on a trusted LAN. Do not bind it to the public
internet.
## Head Tunables (Node A, NVIDIA Desktop)
File: `head/frank-27b-pp.env`
```env
# Educational tunables for the two-node split. Do not deploy.
MODEL_DIR=/home/dank/models
MODEL_FILE=Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf # ~32 GB
# MUCH smaller: use Q8_0 (29.12 GB) if the split must fit more KV.
# Smaller still: Q6_K ~24 GB leaves several GB more room for KV.
MODEL="$MODEL_DIR/$MODEL_FILE"
# Worker (Strix Halo NUC). Multiple --rpc entries allowed; one here.
RPC_SERVERS=192.168.0.143:50052
# Context. 262144 is the native max for this model. Drop to 200000 if
# memory pressure is too high. This is the single-slot ceiling.
CTX_SIZE=262144
PARALLEL=1
# KV cache on GPU. q8_0 K / f16 V saves memory over f16/f16.
CACHE_TYPE_K=q8_0
CACHE_TYPE_V=f16
# Generation batching. Decode is latency-bound over RPC; keep batch modest.
BATCH_SIZE=1024
UBATCH_SIZE=512
# Compute. Match the physical core count on the head box.
THREADS=7
# Sampling (HauhauCS thinking-general defaults).
TEMPERATURE=1.0
TOP_K=20
TOP_P=0.95
MIN_P=0.0
```
## Head Script (Node A, NVIDIA Desktop)
File: `head/head-nvidia.sh`
```bash
#!/usr/bin/env bash
# Run the split model head on the NVIDIA desktop.
# Educational only. Do not run in production.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck disable=SC1091
source "$SCRIPT_DIR/frank-27b-pp.env"
BIN=/home/dank/llama.cpp/build-pp/bin
PORT=7770
RPC_ARGS=()
for s in $RPC_SERVERS; do RPC_ARGS+=(--rpc "$s"); done
# Pick ONE spec variant by uncommenting the matching pair below.
# --- Variant 1: MTP (recommended by research) ---------------------------
# SPEC="--spec-type draft-mtp --spec-draft-model $MODEL_DIR/Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf --spec-draft-n-max 3"
# --- Variant 2: DFlash (block-diffusion draft) ---------------------------
# SPEC="--spec-type draft-dflash --spec-draft-model $MODEL_DIR/Qwen3.6-27B-DFlash-Q8_0.gguf --spec-draft-n-max 8"
# --- Variant 3: ngram-mod (built-in, no draft file) ----------------------
# SPEC="--spec-type ngram-mod --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 --spec-ngram-mod-n-match 24"
# --- Variant 4: no speculation --------------------------------------------
SPEC="--spec-type none"
exec "$BIN/llama-server" \
--model "$MODEL" \
"${RPC_ARGS[@]}" \
${SPEC} \
--ctx-size "$CTX_SIZE" \
--parallel "$PARALLEL" \
--cache-type-k "$CACHE_TYPE_K" \
--cache-type-v "$CACHE_TYPE_V" \
--batch-size "$BATCH_SIZE" \
--ubatch-size "$UBATCH_SIZE" \
--threads "$THREADS" \
--host 0.0.0.0 --port "$PORT" \
--alias frank-27b-pp
```
### Spec Variant Summary
| Variant | draft model file | --spec-type | Notes |
| ------- | ---------------- | ----------- | ----- |
| MTP | `Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf` (~3B) | `draft-mtp` | Parallel-safe, best in benchmarks |
| DFlash | `Qwen3.6-27B-DFlash-Q8_0.gguf` | `draft-dflash` | Known premature-EOS bug |
| ngram-mod | none (built-in) | `ngram-mod` | Zero VRAM cost |
| DSpark | vLLM only | (not in llama.cpp) | Use vLLM PP instead; see below |
DSpark does not exist in llama.cpp. It is a vLLM sparse-attention draft. If
you want DSpark, use vLLM pipeline parallelism, which has its own limits
(no spec decode with PP as of 0.15.x). The llama.cpp path here covers
MTP, DFlash, and ngram-mod.
## Verify (Node A)
File: `head/verify.sh`
```bash
#!/usr/bin/env bash
# Check the split model is up and that both nodes are in the split.
set -euo pipefail
PORT=7770
# Health must return "ok" after model load (minutes, not seconds).
curl -sf "http://127.0.0.1:$PORT/health"
echo
# Show the device split from the server log.
curl -sf "http://127.0.0.1:$PORT/props" | head -40
# Expected: the model and KV cache distributed across CUDA0 (node A)
# and the remote RPC device (node B).
```
## Context Sizing Math
The model is about 29 to 32 GB. The two machines give about 51 GiB of
usable memory. That leaves about 20 to 22 GiB for KV cache. The
Qwen3.6-27B KV cache costs about 85 to 100 KiB per token at q8_0 K /
f16 V. The budget therefore gives a pool of roughly 200k to 260k tokens.
A single 262144-token slot is the practical ceiling for this split.
To run more context, shrink the model quantization. Q6_K (about 24 GB)
frees several GB for KV. Do not try to run the 96 GiB driver figure on the
Strix Halo as real memory. It is an overcommit ceiling; the physical RAM
is 30 GiB.
## What This Does Not Cover
- The remote draft-model-over-RPC path is not confirmed. The draft stays
on the head.
- `--tensor-split` is available to force a manual layer split. The sample
uses the automatic proportional split.
- RDMA (RoCEv2) transport exists but needs a compatible NIC. It adds a few
percent over TCP.
## See Also
- [[llama.cpp RPC]]
- [[Expected Performance]]
- [[Ethernet Bandwidth Math]]
- [[Qwen3.6 27B Spec Methods]]
serverAppConfigs-docs/LLM/Pipeline Parallelism/index.md
-70
View File
@@ -1,70 +0,0 @@
# Pipeline Parallelism
This section covers running one large model across two machines. It focuses
on pipeline parallelism (layer split) over ethernet. The research answers
what speeds are possible and which tools work. The sample scripts show a
two-node Qwen3.6-27B Q8 split for reading only.
## Terminology
Pipeline parallelism (PP) splits model layers across machines. Each machine
computes its layers and passes hidden states to the next machine. Tensor
parallelism (TP) splits each layer across machines and requires an
all-reduce per layer. PP is the only realistic option over ethernet.
Speculative decoding is a separate feature. It uses a small draft model (or
n-grams) to propose tokens that the main model verifies in one pass. It can
run together with PP in llama.cpp.
## The Two Runtimes
```mermaid
graph LR
subgraph llamaRPC[llama.cpp RPC]
A1[ggml-rpc-server on node B] --> A2[llama-server on node A<br/>--rpc nodeB:50052]
end
subgraph vllmPP[vLLM PP]
B1[vllm serve node 0<br/>tp=1 pp=2 rank 0] <-->|hidden states| B2[vllm serve node 1<br/>tp=1 pp=2 rank 1]
end
```
- [[llama.cpp RPC]] — the working, production-capable path for two
heterogeneous nodes over ethernet.
- [[vLLM Pipeline Parallelism]] — supported but heavier; needs Ray or
multiprocessing and does not compose with speculative decoding yet.
## What the Research Found
- [[Ethernet Bandwidth Math]] — a PP boundary moves only the hidden state
(about 10 KiB per token for a 27B model). 1 GbE supports hundreds of
tokens per second on paper; the real limit is latency and RPC overhead.
- [[Mixed-Vendor Experiments]] — NVIDIA + Strix Halo multi-node inference
works today via llama.cpp RPC with separate CUDA and ROCm builds. The
exact NVIDIA-desktop + Strix-Halo-NUC pair has not been benchmarked with
published numbers yet.
- [[Qwen3.6 27B Spec Methods]] — Q8 GGUF variants, and which speculative
methods (MTP, DFlash, ngram-mod) work with RPC. DSpark is vLLM-only.
- [[Expected Performance]] — estimated tokens per second for a 27B Q8 split
across an RTX 3090 and a Strix Halo NUC over 1/2.5/10 GbE.
## Deployment
- [[Qwen3.8 PP Deployment]] — the real Qwen3.8-27B split: head on Dankclaw
(Strix), worker on Dankserver (RTX 3090), b10433 on both sides, scripts in
`scripts/`, morning-of test runbook. Verified gotchas: no `--device` with
`--rpc`, default split is proportional to reported free memory (pin 1,1),
the iGPU side defaults to the ROCm backend.
## Sample Scripts
- [[Sample Script Set]] — a two-node llama.cpp RPC split of
Qwen3.6-27B-Q8_K_P with MTP, DFlash, and ngram-mod variants, sized for
the largest context that fits. For educational reading only. Do not run
it in production.
## See Also
- [[DFlash DSpark Speculative Decoding]]
- [[frank-zealot]]
- [[Dankserver System Specs]]
- [[Dankclaw System Specs]]
serverAppConfigs-docs/LLM/Pipeline Parallelism/llama.cpp RPC.md
-9
View File
@@ -1,9 +0,0 @@
---
type: reference
title: "llama.cpp RPC"
description: "llama.cpp distributed inference via ggml-rpc-server and the --rpc flag: architecture, CLI, versions, limitations, and RPC-compatible speculative decoding."
tags:
- pipeline-parallelism
- research
timestamp: 2026-08-14
---
serverAppConfigs-docs/LLM/Pipeline Parallelism/vLLM Pipeline Parallelism.md
-9
View File
@@ -1,9 +0,0 @@
---
type: reference
title: "vLLM Pipeline Parallelism"
description: "vLLM multi-node pipeline parallelism: flags, Ray/mp backends, interconnect requirements, PP bubble, and PP + speculative decoding support."
tags:
- pipeline-parallelism
- research
timestamp: 2026-08-14
---
serverAppConfigs-docs/LLM/Quantization Production.md
+22 -12
View File
@@ -1,3 +1,13 @@
---
type: "howto"
title: "Quantization Production"
description: "Making a quantized payload here: the conversion recipe, the scoring gate, and the campaign ladder behind a shipped file."
tags:
- llm
- quantization
timestamp: 2026-10-19
---
# Quantization Production — making our own quants, per card and runtime
**Research page (2026-10-08).** Question: how do WE produce quantizations
@@ -73,18 +83,18 @@ Laws that steer everything here:
### Ablation / abliteration (directional ablation)
- Arditi et al. 2024, arXiv 2406.11717: refusal is mediated by a single
direction; orthogonalizing weights against it removes refusals with a
rank-1 edit. Open implementations: `p-e-w/heretic` (automated, above),
rank-1 edit. Open implementations: `p-e-w/heretic` (automated, above)
Artio/abliteration-toolkits, `EleutherAI/repeng`-style rep-overlap
variants **[verified (tool existence: heretic; rest survey)]**.
- Hardware-agnostic fp-editing. On a MoE the edit touches attention/norm
weights, so it composes with any expert quantization.
- Fleet status: frank-dragoon runs an abliterated base **by Frank's choice**
(AGENTS: censored-base refusals were the accepted trade of the 27B era);
(AGENTS: censored-base refusals were the accepted trade of the 27B era)
frank-archon's ukisai Swift1.5 is an abliterated finetune of Qwen3.8.
### HauhauCS QX_K_P "Perfect" quants
- Our frank-zealot publisher. Method, from the publisher's own cards
**[verified]**: (1) abliterate + finetune ("Aggressive"/"Balanced"),
**[verified]**: (1) abliterate + finetune ("Aggressive"/"Balanced")
(2) quantize with **imatrix calibration on the ABLITERATED weights**,
(3) K_P = a **per-model, per-tensor mixed-precision profile** —
"model-specific analysis to selectively preserve quality where it
@@ -194,8 +204,8 @@ u2 draft — all recipe-expressible (see the w4dn8 runbook).
#### Recipe portability: the w4dn8 shape onto other MoEs (2026-10-08 audit)
The pin 1.0.13 container ships SIX arch plugins
(`/opt/radiance/share/radiance/architectures/`): `qwen4exp_fp8`,
**`qwen35moe_fp8`**, `qwen35_fp8`, `qwen35_bf16`, `llama_fp8`,
(`/opt/radiance/share/radiance/architectures/`): `qwen4exp_fp8`
**`qwen35moe_fp8`**, `qwen35_fp8`, `qwen35_bf16`, `llama_fp8`
`llama_dense_fp8` (census via `podman run --rm --entrypoint sh <image> -c`
— the image has no `ls`/`strings`; grep the `.so` files directly, and
the image lives in ROOTLESS storage as `docker.io/stilldeadcode/radiance`).
@@ -204,7 +214,7 @@ the image lives in ROOTLESS storage as `docker.io/stilldeadcode/radiance`).
calibration→GPTQ path exists for the qwen3.5-MoE family; GPTQ itself
lives in `rad-convert`, not the plugin (0 hits in both `.so`).
Worked example — `Qwen/Qwen3.6-35B-A3B` (config: `qwen3_5_moe_text`,
Worked example — `Qwen/Qwen3.6-35B-A3B` (config: `qwen3_5_moe_text`
40 layers, `full_attention_interval=4` DeltaNet hybrid, **256 experts
top-8**, moe_intermediate 512, hidden 2048, MTP ×1, 262k ctx, vocab
248320, multimodal wrapper with vision tower):
@@ -248,13 +258,13 @@ top-8**, moe_intermediate 512, hidden 2048, MTP ×1, 262k ctx, vocab
- EXL3: github.com/turboderp-org/exllamav3 (QTIP lineage; CUDA; Ampere
decode incl. 3090 documented in v1.0 performance write-ups).
- Ampere capability edges: vLLM FP8 docs ("only Hopper and Ada …
W8A8; Ampere supported for W8A16 weight-only utilizing Marlin"),
vLLM Supported-Hardware matrix (Marlin GPTQ/AWQ on SM 8.0/8.6),
W8A8; Ampere supported for W8A16 weight-only utilizing Marlin")
vLLM Supported-Hardware matrix (Marlin GPTQ/AWQ on SM 8.0/8.6)
TensorRT-LLM Support Matrix (FP8 and INT4-AWQ-with-FP8-acts need
SM≥89; NVFP4 Blackwell), SGLang issue #12887 (Ampere FP8-via-Marlin).
- Repo: `scripts/convert-swift15-w4dn8.sh`,
`scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`,
`scripts/kld-gate-swift15.sh`, `scripts/build-llama-rdna4.sh`,
- Repo: `scripts/convert-swift15-w4dn8.sh`
`scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`
`scripts/kld-gate-swift15.sh`, `scripts/build-llama-rdna4.sh`
`scripts/build-kairic-edge-gfx1151.sh`; endpoints [[frank-archon]],
[[frank-dragoon]], [[frank-zealot]]; method [[qwen3.8-strix-optimizations]]
, ; method
and [[GGUF Compatibility]].
serverAppConfigs-docs/LLM/Quantization by Hardware Architecture.md
+1 -4
View File
@@ -132,15 +132,12 @@ quantizer exists for it ([[Quantization Production]]).
|---------|--------|-----|
| 1 | `.hgn` (deployed) | the halogen engine's own format; production door is closed |
| 2 | GGUF K/I quants, K_P blends | the only self-produced option on this card ([[Quantization Production]]) |
| 3 | INT4 W4A16 (Quark) / MXFP4 (Quark) | the retired Ling-3.0-flash vLLM era, measured under the firmware-VGM frame buffer: [[ling-3.0-flash-trial]] |
| 3 | INT4 W4A16 (Quark) / MXFP4 (Quark) | the retired Ling-3.0-flash vLLM era, measured under the firmware-VGM frame buffer: |
| 4 | FP8 (vLLM) | software emulation; memory savings only; no compute win |
## See Also
- [[DFlash DSpark Speculative Decoding]]
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
- [[Running Qwen3.6 27B FP8 on Dankclaw]]
- [[Dankclaw System Specs]]
- [[Dankserver System Specs]]
- [[VLLM Overview]]
- [[Shared Memory Architecture Dankclaw]]
serverAppConfigs-docs/LLM/RAM CPU Offloading in VLLM.md
-206
View File
@@ -1,206 +0,0 @@
---
type: reference
description: "vLLM uses CPU RAM and swap space to extend GPU memory capacity."
tags:
- llm
- vllm
timestamp: 2026-08-28
---
# RAM CPU Offloading in vLLM
## Overview
Stock vLLM offloads **KV pages**, not weights: the scheduler moves KV cache between GPU
and CPU. That is the upstream behaviour and it is NOT the whole story on our hosts — the
tcclaviger/R4D fork ran a real **weight** offload (`--enable-expert-offload`, tiered
expert placement, a pinned host pool per rank), which is how a 147 GiB MoE served from a
246 GiB box: [[RDNA4 MoE Expert Offload VRAM Budget]] and the tcclaviger era row of
[[frank-archon]]. First-hand in-repo evidence outranks a generic upstream claim about our
own hosts ([[Model Swap and Rollback]] §10).
> **Verified 2026-10-19 against `vllm/config/cache.py` on `main`:** there is no
> `swap_space` field — `--swap-space` is gone from current engine args. CPU KV offload is
> `--kv-offloading-size`, and it activates **only** together with `--kv-offloading-backend`
> (default `native`); both fields live in that one file. This page is the fleet's owner of
> that knob contract — other pages link here instead of restating it.
## How CPU Offloading Works
VLLM manages KV cache as pages. Each page holds attention key and value tensors for a set of tokens. When GPU memory is full, VLLM can move pages to CPU memory or disk swap space.
The KV cache swap mechanism works as follows:
1. VLLM allocates GPU blocks for active KV cache pages
2. When GPU blocks are exhausted, VLLM swaps inactive pages to CPU
3. Swapped pages are stored in a CPU memory buffer
4. When a swapped page is needed, VLLM loads it back to GPU
5. If CPU buffer is full, pages are written to disk swap space
## Key Parameters
### gpu_memory_utilization
This parameter controls the fraction of GPU memory used for KV cache.
| Value | Description |
|-------|-------------|
| 0.7 | Uses 70% of GPU memory for KV cache |
| 0.8 | Uses 80% of GPU memory for KV cache |
| 0.92 | Uses 92% of GPU memory for KV cache (default) |
| 0.95 | Uses 95% of GPU memory for KV cache |
Lower values leave more GPU memory for model weights. Higher values allow more concurrent requests but increase swap frequency.
### swap_space (removed) / kv-offloading-size + kv-offloading-backend (current)
`--swap-space` no longer exists in current engine args (verified above); the live pair is
`--kv-offloading-size` with `--kv-offloading-backend`, and offload stays off until the size
is set. The table below is the historical `--swap-space` sizing guidance, kept as a sizing
reference only.
| Value | Description |
|-------|-------------|
| 0 | No CPU swap space |
| 4 | 4 GiB swap space (historical default) |
| 8 | 8 GiB swap space |
| 16 | 16 GiB swap space |
| 32 | 32 GiB swap space |
| 64 | 64 GiB swap space |
The swap space acts as an extension of GPU memory. When GPU blocks are full, VLLM moves KV cache pages to swap space.
### max_model_len
This parameter sets the maximum context length.
| Value | Description |
|-------|-------------|
| auto | VLLM calculates optimal length |
| 2048 | 2048 tokens |
| 4096 | 4096 tokens |
| 8192 | 8192 tokens |
| 32768 | 32768 tokens |
Smaller values reduce memory usage but limit context length.
### max_num_batched_tokens
This parameter sets the maximum number of tokens per batch.
| Value | Description |
|-------|-------------|
| 2048 | Default for online serving |
| 4096 | Moderate batch size |
| 8192 | Large batch size |
| 16384 | Very large batch size |
### max_num_seqs
This parameter sets the maximum number of sequences per batch.
| Value | Description |
|-------|-------------|
| 128 | Default for online serving |
| 256 | Large batch |
### kv_cache_dtype
This parameter sets the KV cache data type.
| Value | Description |
|-------|-------------|
| auto | Uses model default dtype |
| fp8 | Uses FP8 for KV cache (reduces memory by 50%) |
| fp8_e5m2 | Uses FP8 E5M2 format for KV cache |
## Memory Configuration Examples
All three examples carry the retired `--swap-space` syntax and are kept as sizing
references; the live knob pair is named above.
### Small Model on Single GPU
```
vllm serve Qwen/Qwen2.5-1.5B-Instruct \
--gpu-memory-utilization 0.9 \
--swap-space 4 \
--max-num-batched-tokens 4096 \
--max-num-seqs 128
```
### Large Model with CPU Offload
```
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 4 \
--gpu-memory-utilization 0.8 \
--swap-space 16 \
--max-model-len 8192
```
### Memory-Constrained GPU
```
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 4 \
--gpu-memory-utilization 0.7 \
--swap-space 32 \
--max-model-len 4096 \
--max-num-batched-tokens 2048
```
## Preemption Modes
VLLM uses two preemption modes when GPU memory is insufficient:
### Recompute Mode (Default in V1)
VLLM recomputes the prefill for preempted requests. This uses GPU compute instead of memory. This is the default mode in VLLM V1.
### Swap Mode
VLLM swaps KV cache pages to CPU and reloads them later. This uses CPU memory instead of compute.
## Monitoring
Watch for preemption warnings:
```
WARNING scheduler.py: Sequence group 0 is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space.
```
If you see frequent preemptions:
1. Increase `gpu_memory_utilization`
2. Raise the KV offload budget (`--kv-offloading-size`, with `--kv-offloading-backend`)
3. Decrease `max_num_seqs`
4. Increase `tensor_parallel_size`
5. Increase `max_model_len` to match actual usage
## Memory Conservation Tips
- Use quantization (FP8, INT8) to reduce model size
- Reduce `max_model_len` to match actual needs
- Reduce `max_num_seqs` to limit concurrent requests
- Disable CUDA graphs with `--enforce-eager` to save memory
- Set `compilation_config.cudagraph_capture_sizes` to smaller values
- Set `--kv-cache-memory-bytes` to pin the KV budget instead of letting the profiler guess
(the flag our own `scripts/patches/gpu_worker-kv-cap.patch` reads)
- Limit multi-modal inputs with `limit_mm_per_prompt`
## Environment Variables
| Variable | Description | Default |
|----------|-------------|---------|
| vLLM_CPU_KVCACHE_SPACE | KV cache space in GiB for CPU backend | 4 |
| vLLM_CPU_OMP_THREADS_BIND | CPU thread binding for OpenMP | auto |
| vLLM_CPU_NUM_OF_RESERVED_CPU | Reserved CPU cores per rank | 0-1 |
| vLLM_CACHE_ROOT | torch.compile cache directory | ~/.cache/vllm |
## See Also
- [[VLLM Overview]]
- [[VLLM Index]]
- [[Hybrid CPU GPU Processing Dankserver]]
- [[Shared Memory Architecture Dankclaw]]
serverAppConfigs-docs/LLM/RDNA4 Container Serving.md
-258
View File
@@ -1,258 +0,0 @@
---
type: reference
title: RDNA4 Container Serving
description: Bringing a model larger than VRAM onto Dankserver's 2x R9700 (gfx1201) with the tcclaviger vLLM stack — quant-class gating, fetch verification, container GPU-init triage, memory budgeting and the expert-offload plan.
tags: [rdna4, gfx1201, r9700, vllm, expert-offload, podman, rocm]
timestamp: 2026-10-05T00:00:00-07:00
---
# RDNA4 Container Serving
Bring-up path for Dankserver (2x Radeon AI PRO R9700, gfx1201 — card count,
per-card VRAM and host RAM are [[Dankserver System Specs]]'s) with
`docker.io/tcclaviger/vllm` (vLLM 0.29.0.dev, GLM/Qwen-specialised RDNA4 build
with native MoE quant kernels and a first-class expert offload planner). The
tuning history and current deployed shape live in
[[frank-archon]]; this page is the bring-up law.
## 0. Know the box — and its phantom third GPU
`torch.cuda.device_count()` returns **3**; summing `total_memory` overstates
the box as 186.7 GiB. The third device is the 7700X's integrated Vega
(gfx1036): torch reports ~123 GiB for it, `rocm-smi` reports 0.50 GiB,
allocating on it moves system `MemAvailable` 1:1 (it is GTT over DDR5, not
dedicated VRAM), and it dies with `hipErrorInvalidKernelFile` when the image
ships no code for that arch. **Never sum device `total_memory`; never count
the iGPU as headroom or a workload.**
## 1. Preflight: is the quant class EXECUTABLE, not merely present?
Expert offload is mandatory when weights exceed VRAM, and it requires the
checkpoint's experts class to have BOTH a compiled kernel AND an
`offload_recipe` ("a pinned expert has no host copy, so there is no
read-through fallback" — startup aborts otherwise). Check BEFORE downloading
anything:
```bash
podman run --rm --device /dev/kfd --device /dev/dri --device /dev/mem \
-e HIP_VISIBLE_DEVICES=0,1 --entrypoint bash $IMAGE -c '
nm -D /app/r4dhip/r4d.so | grep -oE "gemm_moe_[a-z0-9_]+" | sort -u
grep -rl offload_recipe /opt/vllm/lib/python*/site-packages/vllm/model_executor/layers/fused_moe/'
```
| quant | kernel in `r4d.so` | experts class | verdict |
|---|---|---|---|
| FP8 | yes | `Fp8HipMoEExperts` | **serves** |
| MXFP4 / DSFP4 | yes (`gemm_moe_mxfp4a8_*` / `dsfp4a8_*`) | `R4dMxfp4MoEExperts` / `R4dDsfp4MoEExperts` | **serves** |
| W4A16 | **no** `gemm_moe_w4a16_*` | `R4dW4a16MoEExperts` exists | silently falls back to generic `TritonWNA16Experts`, no recipe → startup halts |
| RFA | none | no class at all | out |
Also confirm the architecture routes natively with NO overlay
(`ModelRegistry.get_supported_archs()` for `Qwen4Exp`/`Glm5`). The stock image
natively carries Qwen3.8/Qwen4Exp and GLM-5.x including MTP heads. **Do not
apply vendor overlays** against a fast-moving pre-release image — the breakage
misleads (a missing symbol in the overlay's OWN copy of a module masks the
real cause, then moves to another symbol each patch;
`cannot import name 'checkpoint_has_lm_head'` = overlay clobbered
`transformers_utils/config.py`). The image ships quantizers for
`rfa|rfi|mxfp4|mxfp4ocp|paro5` under `/app/tools/quantize.py`
(`--fp8-attn --quant-only --dry-run` first) if a re-quant is needed — note it
can run MXFP4/DSFP4/FP8 but only PRODUCE rfa/rfi/mxfp4/paro.
## 2. Fetch with verification, always
Gated repos: token never in argv (`pgrep -a curl` prints command lines).
Build a mode-600 curl config once:
```bash
{ printf 'header = "Authorization: Bearer '; cat ~/.cache/huggingface/token; printf '"\n'; } \
> ~/.cache/huggingface/curl.conf && chmod 600 ~/.cache/huggingface/curl.conf
curl -K ~/.cache/huggingface/curl.conf -f -sS -L -C - -o "$out" "$url"
```
`--fail` is MANDATORY — `curl -o FILE URL` exits 0 on HTTP 401/403 and writes
the error body as a "successful artifact" (a gated repo once yielded three
153-byte shards with identical sha256s and a cheerful DONE banner). `-C -`
resumes torn shards; HTTP/2 cancels leave silently truncated files, so
re-verify EVERY shard's byte count against the Hub manifest (and the LFS
sha256) before serving. The AGENTS.md Secret Rules carry the token-hygiene
law; [[frank-medium-model-swap]] has the same detail for GGUFs and
`.partNofM` fragments.
## 3. Serve — the known-good invocation and its traps
```bash
sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
podman run -d --name <ctr> --ipc=host --ulimit memlock=-1:-1 \
--cap-add SYS_PTRACE \
--device /dev/kfd --device /dev/dri --device /dev/mem \
-e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
-v /home/dank/models/<model>:/models:ro \
$IMAGE /models --served-model-name <name> --port 7770 \
--tensor-parallel-size 2 --max-model-len 65536 \
--kv-cache-memory-bytes 2147483648 \
--max-num-seqs 4 --gpu-memory-utilization 0.95 \
--enable-prefix-caching --trust-remote-code
```
- **Entrypoint already execs `vllm serve "$@"`** — pass the model path as the
first argument and do NOT pass `serve` (podman then tries to pull `serve`
as an image name). `-e VAR=1` must come BEFORE the image name.
- **Mount the whole `/dev/dri`, never individual render nodes** (§4).
- `--ipc=host` and `--shm-size` are mutually exclusive in podman.
- `podman run -d` without `-a` sends container stdout nowhere — read
`podman logs <ctr>`; and vLLM writes `Application startup complete` to
**stderr**, so a `podman logs X 2>/dev/null | grep -q` readiness gate never
fires.
- Start at 65536, not the native ceiling; prove the plan, then scale.
- Load takes ~5 min with `/health` refusing the whole time — do not kill the
process (deploy.sh waits 20 min).
## 4. Container GPU-init triage (ordered)
1. **Enumerate-vs-context trap (most common):** individual
`renderD*` nodes give HIP the render nodes WITHOUT the primary `card*`
nodes — it enumerates agents (`device_count()` returns N) and then refuses
every context: `RuntimeError: No CUDA GPUs are available` on the first
real op. A/B-confirmed 2026-09-19. Rule: `--device /dev/kfd --device
/dev/dri --device /dev/mem`; restrict GPUs with `HIP_VISIBLE_DEVICES`,
never with device nodes.
2. **`device_count()` is not evidence.** Prove usability with a real op
(`(a@a).sum().item()` on a fresh `torch.randn`), then repeat via
`mp.spawn`, `mp.fork`, and `subprocess.run([sys.executable, ...])`, and
cross-check `rocm-smi --showmeminfo vram`.
3. **memlock rlimit → a MISLEADING CUDA OOM.** Host default 8192 KB; a pin
larger than the limit fails as
`torch.AcceleratorError: CUDA error: out of memory` raised from
`pin_memory()` — it names the accelerator but the fault is host pinned
memory. It cannot be raised from inside a container or even from root's
own shell (`Operation not permitted`); only PAM applies it at login, so
fixes only affect NEW sessions:
`/etc/security/limits.d/99-memlock.conf` with
`* - memlock unlimited` + `root - memlock unlimited`; the container needs
`--ulimit memlock=-1:-1` to inherit. Discriminating test after the fix:
the container reports `unlimited` AND small pins succeed; if pins still
fail, memlock was never the cause.
4. **`/proc/meminfo` TAB trap:** the key/value separator is a TAB, so
`grep MemAvailable /proc/meminfo | tr -s " " | cut -d" " -f2` returns
EMPTY and a following awk prints `0.0` — a false "host out of memory" on a
host with hundreds of GiB free, which invites aborting something healthy.
Tab-safe: `awk -v k='MemAvailable:' '$1==k {printf "%.1f", $2/1048576;
exit}' /proc/meminfo`, cross-check `<= MemTotal`, refuse to print a zero.
Use `scripts/dankstatus.sh` instead of hand-rolling.
5. **Silent-parse traps:** `grep -c PATTERN file || echo 0` yields TWO lines
(grep prints 0 AND exits 1) — `[: 0\n0: integer expected`; use
`| head -1` or `|| true`. `systemctl is-enabled` can emit multiple lines.
`rocm-smi --showmeminfo vram --csv` column order is `device, VRAM Total
Memory (B), VRAM Total Used Memory (B)` — reading $3 as total/$4 as used
yields 0.00; filter to `^card[0-9]+$`.
6. **Guard the box:** long loads with co-tenants can drive RAM to 0 — abort
before the OOM killer (`MemAvailable < 4 GiB → podman rm -f`, exit 1).
7. **Drop page cache before a large load** (`echo 3 > /proc/sys/vm/drop_caches`)
and confirm the drop was actually in effect for the run before treating a
failure as a capacity wall — a stale "it doesn't fit" from a contaminated
run is expensive to unwind.
## 5. Memory budgeting: the levers are not where they look
Every figure in this section is a tcclaviger-era boot on the port that is
[[frank-archon]]'s today; the laws they teach are general. The planner builds
`budget = total × util`, subtracts a KV reserve, and gives
the REMAINDER to the expert arena. Consequences, all measured:
- **`--gpu-memory-utilization` grows the ARENA, not the KV pool** (KV stayed
0.64 GiB at util 0.95 and 0.98). The arena is the only elastic term.
- **Lowering `--max-model-len` is the WRONG lever and backfires:** a smaller
context shrinks the reserve, handing the space to the arena — measured 65536
→ 0.64 GiB KV, 32768 → 0.34 GiB KV.
- **Hybrid-attention undercount:** the planner's per-token KV figure for
Qwen4Exp is an undercount — the printed figure and the true one are both on
[[frank-archon]] — because it misses the large fixed per-sequence Mamba state
(~2x), so the reserve is always short and the arena silently covers it.
Fix by pinning the pool: `--kv-cache-memory-bytes <BYTES>` (era reading: a
2 GiB pin = 113,595 tokens ≈ 3.47x concurrency at 32k — that pair is this
page's record of the tcclaviger boots, not [[frank-archon]]'s). For long
context,
`--kv-offloading-size` (GiB) spills KV to host.
- **`--expert-offload-mem` caps the HOST side, not the VRAM arena**
(`cap = int(x * GIB)` against `host_memory_available()`). Do not reach for
it to fix a VRAM problem; to grow the arena, shrink the KV pin or raise
util (see [[frank-archon]] for the
util=1.0 unbootability law).
**Read the plan — it is the real acceptance test:**
```
expert offload: AUTO-ENABLED, weights 62.54 GiB per rank exceed the 30.27 GiB budget (TP2)
budget (total x util) 30.27 GiB
non-expert weights 6.28 GiB
KV reserve 0.36 GiB <- sanity-check against the model's own per-token figure
CUDA graphs 0.66 GiB
runtime overhead 7.60 GiB
expert cache space 15.36 GiB 6712 slots of 2.344 MiB
host-backed experts/rank 52.74 GiB 23040 experts
host: 105.48 GiB across 2 ranks of 160.89 GiB usable
```
Host fits when `host_total < usable`. The PLE n-gram table rides memfd and is
SHARED across TP ranks.
**Loading mechanism** (for reasoning about peak RAM): weights never materialise
in VRAM — expert Parameters start as CPU staging tensors; each layer's
`weight_loader` counts shard events and, on layer completion,
`ExpertTopology.split_layer` moves that layer's experts into the VRAM arena
and compacts the rest into exact-size pinned host tensors behind UVA. Peak
host RAM = `host-backed set + layers still loading`, not the whole model. At
runtime `apply_hot_cold` sends arena-resident experts down a hot GEMM and the
rest straight over PCIe via UVA views — no staging copy in the critical path.
**Emulation check:** a dequantize-to-compute path doubles real host RAM — the
measured pair (offload request vs host footprint) is on [[frank-archon]] — and
the doubling is invisible in the weight count. Read the log for `Using
'EMULATION' Mxfp4 MoE backend` before concluding "doesn't fit".
## 6. Performance profile and honest measurement
Decode on this pair is **PCIe-gather-bound, context-independent**: the rate is
flat from a few hundred to sixteen thousand context tokens, so KV attention is
not the cost (the flat figure and its MTP/gather deltas are
[[Tcclaviger Fork Ops]]'s profile of the same shape). With ~29 % of experts
arena-resident, under a gigabyte crosses PCIe per token at the engine's
effective read rate; cold-vs-warm differing ~10 % confirms the LRU works;
growth is linear in hit rate, and only speculative decode cuts sequential STEPS
rather than making each cheaper.
**Benchmark traps that produced fake numbers here:**
- `max_tokens=1` is not TTFT (the server can satisfy it without doing the
work — prefill looked flat at 1.26 s for 16k AND 30k prompts). A 64-token
non-streamed reply is time-to-64-tokens (decode-bound, flat ~9.6 s at every
length). Correct TTFT: stream and time the FIRST content-bearing chunk.
- A small repeating word loop is prefix-cacheable AND n-gram-predictable — a
43-word cycle reported 22,897 tok/s prefill, almost entirely cache hit. Use
a large vocabulary in shuffled order and print `cached_tokens`.
- ~1.2 s fixed per-request overhead swamps prefill below ~16k tokens; compare
prefill at 30k+ only.
- Size prompts with `/v1/messages/count_tokens` (a chat call for counting
pollutes the metric); guessing words-per-token silently overran the window
(HTTP 400).
- Always run a canary: plant a unique token at the END of a long prompt and
ask the model to quote it — proves prompts reach the model at all.
- Measure warm AND long (`podman exec` + urllib at 127.0.0.1:7770): a 33-token
reply read 14.6 cold / 20.3 warm; a 768-token generation hit 25.7; two
concurrent streams scaled to 47 aggregate.
**Levers (all opt-in):** `--expert-prefetch N` takes an INT depth, not a
boolean (default 0; at depth 1 only ~10 % cold-vs-warm difference — the cache
already works); `CLAV_TUNABLEOP_SWEEP=1` autotunes GEMM shapes (silently
skipped under expert offload unless `CLAV_TUNABLEOP_WITH_OFFLOAD=1` too —
see [[frank-archon]] for why it was rejected);
`--expert-precise-memory` swaps the fitted overhead figure in.
## Related
- [[frank-archon]] — deployed shape, plan law,
measured campaigns
- [[PCIe Link and Atomics]] — why decode is gather-bound and what cannot fix it
- [[RDNA4 MoE Expert Offload VRAM Budget]] — arena/plan arithmetic deep-dive
- [[frank-dragoon]] and [[frank-zealot]] — window discipline on live hosts
serverAppConfigs-docs/LLM/RDNA4 MoE Expert Offload VRAM Budget.md
+4 -5
View File
@@ -1,7 +1,7 @@
---
type: reference
title: RDNA4 MoE Expert Offload VRAM Budget
description: The expert-offload plan arithmetic on the tcclaviger fork — which terms a deploy can move, the util law, the CUDA-graph charge, what free VRAM really is, knife-edge survivability, parallelism ceilings and the context ladder.
description: "Expert-offload arithmetic on the RDNA4 fork: movable terms, the utilisation law, graph charge, ceilings, context ladder."
tags: [vllm, expert-offload, vram, arena, kv-cache, ladder, tcclaviger]
timestamp: 2026-10-05T00:00:00-07:00
---
@@ -14,7 +14,6 @@ figure and rate below is from [[frank-archon]]'s tcclaviger era (2026-09-29 →
to host RAM) — that page owns the endpoint's shape, tuning history and era
record; this page owns only the plan arithmetic, the ladder method and the audit
procedure. Boot-failure capture lives in
[[vLLM Boot Forensics and Config Interrogation]].
## 1. The plan law
@@ -164,7 +163,7 @@ spec maximum, never hand arithmetic.
`Tried to allocate 10.28 GiB … 9.46 GiB free` died in KV ALLOCATION.
6. **Clean between rungs:** stop the transient unit, `podman rm -f`, kill
orphans by explicit PID
(`ps aux --sort=-rss | awk '/vllm\/bin\/python3/ && $6 > 50000000 {print $2}'`),
(`ps aux --sort=-rss | awk '/vllm\/bin\/python3/ && $6 > 50000000 {print $2}'`)
wait for `MemAvailable` to recover.
**The hazard that cost production:** a leftover ladder script whose teardown
@@ -215,7 +214,7 @@ at graph capture; 6 seqs fp8 → 10.70 GiB ask with 9.18 GiB free (fails) —
## Related
- [[RDNA4 Container Serving]] — the bring-up path and container traps
- [[vLLM Boot Forensics and Config Interrogation]] — capture and contracts
- the bring-up path and container traps
- capture and contracts
- [[frank-archon]] — measured history
- [[Llama Cpp Serving Sizing]] — the llama.cpp equivalent
serverAppConfigs-docs/LLM/Running Qwen3.6 27B FP8 on Dankclaw.md
-246
View File
@@ -1,246 +0,0 @@
---
type: reference
description: "Qwen3.6-27B is a dense vision-language model from the Qwen3.6 series."
tags:
- llm
timestamp: 2026-09-03
---
# Running Qwen3.6 27B FP8 on Dankclaw
> **RETIRED - vLLM era (before the 2026-08-17 redo).** Nothing on this page serves
> today: no vLLM or ROCm process runs on Dankclaw, and the iGPU endpoint at
> `192.168.0.143:7770` is [[frank-dragoon]] (halogen flash-server). The page stays as
> the era record of the Qwen3.6-27B-FP8 vLLM bring-up and the Strix Halo ROCm
> recipes ([[frank-redo-2026-08-17]]). The hardware table below carries the era's
> host figures; live ones are [[Dankclaw System Specs]].
## Model Information
Qwen3.6-27B is a dense vision-language model from the Qwen3.6 series. It uses 27B parameters with gated DeltaNet and attention layers.
| Property | Value |
|----------|-------|
| HuggingFace ID | `Qwen/Qwen3.6-27B-FP8` |
| Base Model | `Qwen/Qwen3.6-27B` |
| Architecture | Qwen3_5ForConditionalGeneration |
| Total Parameters | 27B |
| Hidden Size | 5120 |
| Layers | 64 (16 blocks of 3 DeltaNet + 1 Attention) |
| Attention Heads | 24 Q, 4 KV (head dim 256) |
| DeltaNet Heads | 48 V, 16 QK (head dim 128) |
| Intermediate Dim | 17,408 |
| Vocab Size | 248,320 (padded) |
| Context Length | 262,144 (natively extensible to 1,010,000) |
| Quantization | FP8 (block size 128) |
| Vision Encoder | Yes (image-text-to-text) |
## Hardware: Dankclaw
*Era figures (2026-08). The host has changed since - [[Dankclaw System Specs]].*
| Property | Value |
|----------|-------|
| CPU | AMD RYZEN AI MAX+ 395 w/ Radeon 8060S (Strix Halo) |
| GPU | Integrated Radeon 8060S (unified memory) |
| Physical RAM | 64 GiB (61 GiB visible; corrected 2026-08-17) |
| GPU-Visible (ROCm) | 96 GiB (overcommitted) |
| Swap | 8 GiB |
| ROCm | 7.2.4 |
| Python | 3.12.13 (vLLM venv) |
## Memory Calculation
| Component | BF16 | FP8 |
|-----------|------|-----|
| Model weights (27B) | 54 GB | 13.5 GB |
| Vision encoder | ~1 GB | ~0.5 GB |
| **Total weights** | ~55 GB | ~14 GB |
| KV cache (64K ctx, 1 seq) | ~1 GB | ~0.5 GB |
| Overhead | ~2 GB | ~2 GB |
| **Total (approximate)** | ~58 GB | ~16.5 GB |
The FP8 variant requires approximately 16-17 GB. ThinkingCap-FP8 requires ~29 GB. Both fit within the 61 GiB physical RAM, but ROCm reports 96 GiB to vLLM. Set `--gpu-memory-utilization` conservatively (0.80) to avoid OOM.
## Prerequisites
> **Era note (2026-08, superseded twice):** at writing time Dankclaw's primary
> service was the vLLM-ROCm Ling-3.0-flash CIRU INT4 trial ([[ling-3.0-flash-trial]],
> retired 2026-08-17), and this page documented the alternative vLLM FP8 deployment
> of Qwen3.6-27B. Both paths are dead; `scripts/frank-dragoon.service` today is the
> halogen flash-server unit ([[frank-dragoon]]).
Install vLLM for ROCm. Version 0.24.0+ is required for Qwen3.6 with speculative decoding:
```
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
```
Installed: vLLM 0.26.0 in Python 3.12.13 venv at `/home/dank/vllm-rocm-venv`.
### MPI Dependency
Torch requires `libmpi_cxx.so.40`. If missing, install `openmpi-bin`:
```
sudo apt install -y openmpi-bin
```
### Environment Variables
Set these for ROCm on Strix Halo:
```
export HSA_OVERRIDE_GFX_VERSION=11.5.1
export HF_HOME=/home/dank/.cache/huggingface
```
## vLLM Serve Commands
### Text-Only (Recommended for Memory Savings)
Skip the vision encoder to reduce memory:
```
vllm serve Qwen/Qwen3.6-27B-FP8 \
--port 8000 \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.85 \
--swap-space 8 \
--max-model-len 131072 \
--max-num-batched-tokens 2048 \
--max-num-seqs 32 \
--reasoning-parser qwen3 \
--language-model-only \
--dtype auto
```
### With Vision Encoder
```
vllm serve Qwen/Qwen3.6-27B-FP8 \
--port 8000 \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.85 \
--swap-space 8 \
--max-model-len 131072 \
--max-num-batched-tokens 2048 \
--max-num-seqs 32 \
--reasoning-parser qwen3 \
--dtype auto
```
### Parameters Explained
| Parameter | Value | Reason |
|-----------|-------|--------|
| --kv-cache-dtype fp8 | FP8 KV cache | Cuts KV cache memory by 50% |
| --gpu-memory-utilization 0.85 | 85% of unified memory | Leaves 15% for CPU/OS |
| --swap-space 8 | 8 GB CPU swap | Matches system swap size |
| --max-model-len 131072 | 128K context | Full 262K is large but possible with 61 GB |
| --language-model-only | Skip vision | Saves ~1 GB for KV cache |
| --reasoning-parser qwen3 | Qwen3 parser | Required for thinking mode |
## Sampling Parameters
| Mode | Temperature | Top P | Top K | Min P | Presence Penalty |
|------|-------------|-------|-------|-------|-----------------|
| Thinking (general) | 1.0 | 0.95 | 20 | 0.0 | 0.0 |
| Thinking (coding) | 0.6 | 0.95 | 20 | 0.0 | 0.0 |
| Non-thinking | 0.7 | 0.80 | 20 | 0.0 | 1.5 |
Set `max_tokens` to 32768 for general use. Set to 81920 for complex reasoning.
## API Examples
### Chat with Thinking
```
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3.6-27B-FP8",
"messages": [{"role": "user", "content": "Solve: 2+2"}],
"max_tokens": 32768,
"temperature": 1.0,
"top_p": 0.95,
"top_k": 20
}'
```
### Non-Thinking Mode
```
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3.6-27B-FP8",
"messages": [{"role": "user", "content": "Hello"}],
"max_tokens": 2048,
"temperature": 0.7,
"top_p": 0.8,
"top_k": 20,
"presence_penalty": 1.5,
"extra_body": {
"chat_template_kwargs": {"enable_thinking": false}
}
}'
```
### Tool Calling
```
vllm serve Qwen/Qwen3.6-27B-FP8 \
--port 8000 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder
```
## Ultra-Long Context (YaRN)
Extend context beyond 262,144 tokens:
```
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Qwen/Qwen3.6-27B-FP8 \
--hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}' \
--max-model-len 524288
```
## ROCm Notes
- Strix Halo (gfx1151) is listed as a supported AMD GPU for vLLM
- FP8 uses **software emulation** on RDNA architecture (gfx1151 has no native FP8 tensor cores). This gives memory savings, not full compute speedup. Compare against a BF16 test first
- Attention backends: try `ROCM_AITER_UNIFIED_ATTN` if the default backend underperforms
- Context length on ROCm may be limited for some models. Start with 131072 and increase gradually
## Troubleshooting
### Out of Memory
1. Use `--language-model-only` to skip vision encoder
2. Reduce `--gpu-memory-utilization` to 0.75
3. Reduce `--max-model-len` to 65536
4. Reduce `--max-num-batched-tokens` to 1024
### FP8 Errors
1. Try BF16 first: `vllm serve Qwen/Qwen3.6-27B --dtype bfloat16`
2. Verify ROCm version supports FP8
3. Set `HSA_OVERRIDE_GFX_VERSION=11.5.1`
4. Use `--kv-cache-dtype auto` instead of fp8
### Slow Inference
1. Check GPU utilization with `rocm-smi`
2. Verify attention backend is optimal
3. Increase `--max-num-batched-tokens` if memory allows
4. Ensure CPU cores are not throttled
## See Also
- [[VLLM Overview]]
- [[Dankclaw System Specs]]
- [[Shared Memory Architecture Dankclaw]]
- [[RAM CPU Offloading in VLLM]]
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
serverAppConfigs-docs/LLM/Running Qwen3.6 35B A3B FP8 on Dankserver.md
-210
View File
@@ -1,210 +0,0 @@
---
type: reference
description: "Qwen3.6-35B-A3B is a Mixture-of-Experts vision-language model from the Qwen3.6 series."
tags:
- llm
timestamp: 2026-09-03
---
# Running Qwen3.6 35B A3B FP8 on Dankserver
> **RETIRED.** This page documents the FP8 vLLM path for Qwen3.6-35B-A3B on Dankserver. The llama.cpp 35B service is retired (2026-08-16), its unit file was removed (2026-08-17), and the FP8 vLLM unit (`vllm-qwen36`) never shipped and is disabled. The content is kept as research history. The RTX 3090 now runs the PP worker: see [[Qwen3.8 PP Deployment]].
## Model Information
Qwen3.6-35B-A3B is a Mixture-of-Experts vision-language model from the Qwen3.6 series. It uses 35B total parameters with only 3B activated per token.
| Property | Value |
|----------|-------|
| HuggingFace ID | `Qwen/Qwen3.6-35B-A3B-FP8` |
| Base Model | `Qwen/Qwen3.6-35B-A3B` |
| Architecture | Qwen3_5MoeForConditionalGeneration |
| Total Parameters | 35B |
| Activated Parameters | 3B |
| Experts | 256 (8 routed + 1 shared per token) |
| Layers | 40 (10 blocks of 3 DeltaNet + 1 Attention) |
| Attention Heads | 16 Q, 2 KV (head dim 256) |
| DeltaNet Heads | 32 V, 16 QK (head dim 128) |
| Expert Intermediate Dim | 512 |
| Vocab Size | 248,320 (padded) |
| Context Length | 262,144 (natively extensible to 1,010,000) |
| Quantization | FP8 (block size 128) |
| Vision Encoder | Yes (image-text-to-text) |
## Hardware: Dankserver
| Property | Value |
|----------|-------|
| GPU | NVIDIA GeForce RTX 3090 |
| VRAM | 24 GB |
| System RAM | 62 GB |
| Swap | 127 GB |
## Memory Calculation
| Component | BF16 | FP8 |
|-----------|------|-----|
| Activated weights (3B) | 6 GB | 1.5 GB |
| Inactive experts (32B) | 64 GB | 16 GB |
| Vision encoder | ~1 GB | ~0.5 GB |
| **Total weights** | ~71 GB | ~18 GB |
The FP8 variant requires approximately 18 GB for weights. This leaves approximately 6 GB of the RTX 3090's 24 GB VRAM for KV cache and overhead. Use aggressive memory settings.
## Prerequisites
Install vLLM. Version 0.19.0 or later is required for Qwen3.6:
```
uv pip install vllm --torch-backend=auto
```
## Hardware Quantization Notes (RTX 3090 / Ampere)
The RTX 3090 is an Ampere GPU (compute capability 8.6). It has **no native FP8 tensor cores**. FP8 W8A8 compute runs through a fallback/dequantization path, so the FP8 model gives memory savings but not full compute speedup. vLLM loads these models and runs them; verify quality against a BF16 run. `--kv-cache-dtype fp8` is still valid on CUDA 12 for an FP8 KV cache.
> Do not run on production without a BF16 comparison. The FP8 precision path may differ from native-FP8 hardware.
## vLLM Serve Commands
### Text-Only (Recommended for Memory Savings)
Skip the vision encoder to free VRAM for KV cache:
```
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
--port 8000 \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.75 \
--swap-space 32 \
--max-model-len 131072 \
--max-num-batched-tokens 2048 \
--max-num-seqs 32 \
--reasoning-parser qwen3 \
--dtype auto
```
### With Vision Encoder
If you need multimodal input, expect higher VRAM usage:
```
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
--port 8000 \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.7 \
--swap-space 32 \
--max-model-len 131072 \
--max-num-batched-tokens 1024 \
--max-num-seqs 16 \
--reasoning-parser qwen3 \
--dtype auto
```
### Parameters Explained
| Parameter | Value | Reason |
|-----------|-------|--------|
| --kv-cache-dtype fp8 | FP8 KV cache | Cuts KV cache memory by 50% |
| --gpu-memory-utilization 0.75 | 75% of VRAM | Leaves room for all MoE weights |
| --swap-space 32 (removed) | 32 GB CPU overflow | Modern vLLM removed swap-space; use --kv-offloading-size |
| --max-model-len 131072 | 128K context | Full 262K context is too large for 24 GB |
| --max-num-batched-tokens 2048 | Conservative batch | Prevents OOM with MoE |
| --reasoning-parser qwen3 | Qwen3 parser | Required for thinking mode output |
### Model Storage
Root disk has only 20 GB free. Store models on /mnt/entropy:
```
export HF_HOME=/mnt/entropy/huggingface
```
### Active Service Conflict
The retired `frank-zealot` service (llama.cpp) used about 22 GB of the 24 GB VRAM. The current PP worker holds ~23.8 GiB. The `vllm-awq-qwen36` (Podman) and `vllm-qwen36` units are disabled. Before starting any research service, verify that the GPU is free:
```
nvidia-smi # verify GPU is free
```
## Sampling Parameters
| Mode | Temperature | Top P | Top K | Min P | Presence Penalty |
|------|-------------|-------|-------|-------|-----------------|
| Thinking (general) | 1.0 | 0.95 | 20 | 0.0 | 0.0 |
| Thinking (coding) | 0.6 | 0.95 | 20 | 0.0 | 0.0 |
| Non-thinking | 0.7 | 0.80 | 20 | 0.0 | 1.5 |
Set `max_tokens` to 32768 for general use. Set to 81920 for complex reasoning.
## API Examples
### Chat with Thinking
```
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3.6-35B-A3B-FP8",
"messages": [{"role": "user", "content": "Solve: 2+2"}],
"max_tokens": 32768,
"temperature": 1.0,
"top_p": 0.95,
"top_k": 20
}'
```
### Non-Thinking Mode
```
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3.6-35B-A3B-FP8",
"messages": [{"role": "user", "content": "Hello"}],
"max_tokens": 2048,
"temperature": 0.7,
"top_p": 0.8,
"top_k": 20,
"presence_penalty": 1.5,
"extra_body": {
"chat_template_kwargs": {"enable_thinking": false}
}
}'
```
## Ultra-Long Context (YaRN)
Extend context beyond 262,144 tokens. Use this environment variable and override:
```
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
--hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}' \
--max-model-len 524288
```
## Troubleshooting
### Out of Memory
1. Kill llama-server to free 20 GB VRAM
2. Use `--language-model-only` to skip vision encoder
3. Reduce `--max-model-len` to 65536
4. Reduce `--max-num-batched-tokens` to 512
5. Reduce `--max-num-seqs` to 8
### Slow Inference
1. Verify GPU is exclusive to vLLM
2. Check temperature with `nvidia-smi`
3. Ensure `--kv-cache-dtype fp8` is set
4. Increase `--max-num-batched-tokens` if memory allows
## See Also
- [[VLLM Overview]]
- [[Dankserver System Specs]]
- [[Hybrid CPU GPU Processing Dankserver]]
- [[RAM CPU Offloading in VLLM]]
- [[Running Qwen3.6 27B FP8 on Dankclaw]]
serverAppConfigs-docs/LLM/Shared Memory Architecture Dankclaw.md
+4 -9
View File
@@ -85,7 +85,7 @@ Available backends for ROCm:
All seven names exist on current vLLM `main` (`AttentionBackendEnum` in
`vllm/v1/attention/backends/registry.py` —
[source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/v1/attention/backends/registry.py),
[source](https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/v1/attention/backends/registry.py)
read 2026-10-18), and `--attention-backend`, `--numa-bind`, `--numa-bind-nodes`
are real `vllm serve` flags (`vllm/engine/arg_utils.py:1098,1171-1176`).
@@ -103,7 +103,7 @@ vllm serve model \
- `gpu_memory_utilization`: Set lower (0.8-0.85) to leave room for CPU operations
- `max_model_len`: Match actual context needs
- `--swap-space` is retired from current vLLM; the CPU KV-offload knob pair, with the upstream `CacheConfig` verification, is owned by [[RAM CPU Offloading in VLLM]]
- `--swap-space` is retired from current vLLM; the CPU KV-offload knob pair, with the upstream `CacheConfig` verification, is owned by
### NUMA Binding
@@ -213,7 +213,7 @@ Unified memory limits concurrent requests:
The two "No" rows (Async Output, Multi-step Decoding) were transcribed from a
vLLM feature-support snapshot and not re-checked against current upstream
*(unverified 2026-10-18)*.
- (unverified 2026-10-18)*.
### Context Length
@@ -226,7 +226,7 @@ FP8 quantization depends on GPU architecture:
- MI300 series supports native FP8
- Radeon RX 7900 supports FP8 via software emulation
- Strix Halo (gfx1151) uses **software emulation** for FP8 matmuls — RDNA 3.5 has
no FP8 tensor instructions ([FP8-ROCM](https://github.com/wabibito/FP8-ROCM));
no FP8 tensor instructions ([FP8-ROCM](https://github.com/wabibito/FP8-ROCM))
expect memory savings, not full compute speedup. (Conflicting vendor-side
evidence exists for an FP8/INT8 WMMA path used by FSR4 on RDNA 3.5, so treat
the matmul claim as the settled half.)
@@ -294,8 +294,3 @@ rocm-smi --showtemperature
## See Also
- [[VLLM Overview]]
- [[VLLM Index]]
- [[RAM CPU Offloading in VLLM]]
- [[Hybrid CPU GPU Processing Dankserver]]
- [[Running Qwen3.6 27B FP8 on Dankclaw]]
serverAppConfigs-docs/LLM/Tcclaviger Fork Ops.md
-202
View File
@@ -1,202 +0,0 @@
---
type: reference
title: Tcclaviger Fork Ops
description: Operating the tcclaviger RDNA4 vLLM fork on dankserver (shape G era, rollback target since the 2026-10-05 FreeToken cutover) — the ad-hoc sweep harness, measured knob verdicts, sweep hygiene, the large-MoE serving recipe, and compile-cache persistence.
tags: [tcclaviger, vllm, dankserver, expert-offload, sweeps, cache]
timestamp: 2026-10-05T00:00:00-07:00
---
# Tcclaviger Fork Ops
The tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`,
vLLM 0.29.0.dev, GLM-5.3/Qwen-specialised with native MoE quant kernels and a
first-class expert-offload planner) served `frank-archon` at shape G until
the 2026-10-05 FreeToken cutover; it remains the rollback target
(`git tag frank-archon-vllm-shapeG-20261003`). Full plan law and VRAM
arithmetic: [[RDNA4 MoE Expert Offload VRAM Budget]]; boot forensics:
[[vLLM Boot Forensics and Config Interrogation]]; window discipline:
[[LLM Bench Windows and Measurement]].
## The ad-hoc sweep harness (dankserver)
Serve script `/home/dank/serve_qwen_fp8.sh` with env hooks: `SPEC` (MTP
depth), `TUNABLE` (TunableOp), `PREFETCH`, `UTIL`, `MAXLEN`, `KV_BYTES`,
`MNBT`, `UVA_TOK`, `GATHER_C/GATHER_L` (R4D_LRU gather), `TELEM`
(R4D_LRU_TELEMETRY), `PROFDIR` (torch profiler bind-mount + --profiler-config
JSON). The script re-copies the harness (`mtp_measure.py`, `bench_iter.py`)
into every fresh container — `podman cp` files die on container recreate.
- Measure ONLY via usage counters: `mtp_measure.py` (3 depths) or
`mtp_measure_short.py` (166 ctx). `decode = completion_tokens/(wall −
ttft)`; chunk counting is INVALID under spec decode.
- Jobs: `bash /home/dank/job.sh start NAME CMD…` / `wait NAME SECS 40` /
`out NAME` ([[Remote Job Runner]]).
- **Validity check on ANY speed result:** `completion_tokens == max_tokens`
(else repetition-halt or early EOS), finish_reason, and inspect the text —
`bench_iter` prompts are repeated word-lists + `!` padding, loop-prone. A
loop makes LRU hits ~100% → PCIe traffic vanishes → fake 3–4× decode.
Acceptance swings 48–61% run-to-run — noise; decode is the metric.
- Each arm = full reload: ~8.5 min cold, ~12 min with measurement.
- Readiness gate: `podman logs <c> 2>&1` (stdout+stderr merged) for
`Application startup complete` — it is on STDERR; an unmerged grep can
never match and burns the whole wait timeout.
- GPU util ~0% during load/compile is NORMAL (weights + CUDA-graph capture
are CPU-side); VRAM climbing is the load indicator.
- Config edit discipline: the serve script is patched by scp'd python
scripts with asserts (anchor must exist, addition must not); `bash -n`
after every patch.
## Measured knob verdicts (2026-09-29 campaign)
Incumbent: `SPEC=4 PREFETCH=1 UTIL=0.98 MAXLEN=32768 KV_BYTES=2147483648
TUNABLE=0`, stock gather 8×16 → 26.80/25.15/26.81 tok/s @ 166/1087/4064 ctx.
| Knob | Verdict | Evidence |
|---|---|---|
| Gather geometry `R4D_LRU_GATHER_{CHUNKS,LANES}` | **RULED OUT** | 16×32 worse (~24); 32×64 tie (26.6). The 9.5 vs 22.7 GB/s read-path gap is per-transfer structure (~105 µs/slab vs 78 ideal), not a tunable knob |
| MTP depth | **RULED OUT** | SPEC=3 ~9% worse; SPEC=6 collapses (13.5–19 tok/s, acceptance 17.4%) — also proves UVA read-through (crosses at step tokens > `R4D_LRU_UVA_TOKENS=5`) is SLOWER than copy-in |
| TunableOp | **REJECTED** | `CLAV_TUNABLEOP_SWEEP=1` silently SKIPPED under expert offload unless `CLAV_TUNABLEOP_WITH_OFFLOAD=1`; adds ~7 min/load, persists nothing (container-local), and CORRUPTS NUMERICS (loops, repetition-halt, fake 79.8 tok/s @ 96.5% acceptance on degenerate completions). Any "3× win" with early-EOS + absurd acceptance = the loop artifact |
| MNBT | neutral at 8192 vs 2048 | small values re-chunk prefill only (hurts TTFT, never decode — decode step = max-num-seqs × (spec+1) tokens). No ubatch concept in vLLM |
Decode on this stack is PCIe-gather-bound (~30% GPU duty cycle), so
batch-shaped compute knobs move ~nothing.
Measured with no spec: **12.11 tok/s FLAT from 304 to 16,095 context tokens** — KV
attention is not the cost, so a context-slope hunt on this stack is null by construction.
The `R4D_*/CLAV_*` knob family (QSA_*, MHC, ROUTE, ACT_QUANT, AR_QUANT,
SILU_QUANT) are kill-switches defaulting ON, not perf dials; real levers are
the R4D_LRU_* family, TunableOp (rejected), R4D_LRU_TELEMETRY, and spec
depth.
Telemetry gotchas: `--profiler-config` takes a JSON blob, not separate
flags; `torch_profiler_with_stack` defaults TRUE — a full measure inside a
profile window OOM-kills the engine worker (~40 GB trace buffer; kernel
kills python3 at ~95 GB anon-rss). Keep windows to ~15 s / one depth
(`mtp_measure_short.py`). `telemetry_report` (`expert_cache_rank<N>.json`)
fires ONLY from the worker's profiler-stop path; worker-side stop can hang
>5 min on ROCm/kineto. No alternative trigger exists. The LRU hit rate
measured 82.45% — NOT the ~27% the arena-fraction arithmetic implies;
temporal reuse dominates, so never infer miss rates from resident-slot
fractions.
## Sweep hygiene (fork-specific preflight)
1. **Knob landed:** for each `-e` env knob, confirm in-container
`podman exec … env` (or the engine log's confirmed-values line). A knob
the fork silently ignores produces a "parity" arm that is really the
default — the v1 gather sweep lost all three arms to exactly this
(GATHER_C/GATHER_L silently dropped, all arms ran stock 8×16).
2. **Prefetch uniformity:** every arm in a sweep MUST carry the same
expert-prefetch setting. The 2026-09-29 campaign measured losing arms
with prefetch ON and incumbents OFF; untaxed, its verdicts inverted
(no-spec 12.11→41.47; gather 32×64 parity→+18.7%). Never re-read a taxed
verdict.
3. **Spec arms must prove engagement:** engine log
`SpecDecoding metrics: Mean acceptance length …` and
`curl :7770/metrics | grep vllm:spec_decode_`. Zero such lines = MTP off.
4. Boot-fail fast-detection: poll `podman ps` INSIDE the health loop; a
refused boot dies in one poll instead of 20 min of health waits. Capture
>8 lines of `podman logs` on failure — the root cause (e.g. a SEQS init
refusal) lives higher up.
5. Report per-depth tables, never one number: g32×64's win grows with depth
(54@1k, 39.5@short) — a single 4064 median hides the short-ctx cost. TTFT
rows are noisy ±0.15 s and non-monotonic across depths; compare same-ctx
rows across arms only.
6. Generate ≥256 tokens per sample (at 8 tokens inter_token is first-token
noise).
Boot-log facts survive measure failures: KV pool size (`GPU KV cache size`),
arena slots (`expert cache: … -> N slots`), and the spec-reservation ladder
(MTP3 766,666 / MTP4 729,545 / MTP6 665,263 tokens) print at BOOT — a
measure-stage bug costs decode tables but not boot-log findings.
**MTP depth-brittleness law (2026-10-02, all arms rc=0):** MTP3 on this fork
is depth-brittle — 47.01 t/s at 166 tokens (47.88 at temp 0.7) but 23.09 at
4064, vs no-spec baseline 37.02 / 33.99. MTP3 + SEQS 16 + MNBT 4096 + util
0.95 + ladder [4..32] = best overall 37.04 / 38.61 t0.7 and keeps 33.15 at
4064. `--expert-precise-memory` is load-bearing for MTP (same arm with it
OFF = worst arm, 19.81 at 4064). Every spec arm costs KV (398,658 → 313,002
tokens). Acceptance is real: mean length 2.4–2.8, per-position
0.76/0.55/0.33, avg 52–57%. Next lever named by the engine: MNBT 4096 is
exactly SEQS×(n+1) (`suboptimal` warning) → try 8192.
## Serving a model larger than VRAM (dankserver recipe)
Dankserver's card count, per-card VRAM and host RAM are in [[Dankserver System Specs]];
`torch.cuda.device_count()` returns 3 on this box and summing `total_memory` overstates it
badly (the Raphael iGPU is in the list — reading discipline: [[Llama Cpp Serving Sizing]]).
Everything else about running this image — the quant-kernel gate that decides
which checkpoints can serve at all, the entrypoint contract, the known-good
invocation, the `/dev/dri` and memlock traps, the util and max-model-len arena
traps, the `--ipc=host` / `-d` stdout traps and the gated-fetch discipline — is
written once on [[RDNA4 Container Serving]], and the plan arithmetic behind the
arena/KV traps is [[RDNA4 MoE Expert Offload VRAM Budget]]'s. Three findings
stay here because this harness produced them:
- **The KV reserve undercounts hybrid-attention models ~2×** (the planner's
B/token figure against ~18,000 real: the Mamba state is a fixed per-sequence
cost). Fix by pinning the pool: `--kv-cache-memory-bytes`.
- **`--expert-offload-mem` caps the HOST side**, not the VRAM arena —
structurally orthogonal to `--expert-precise-memory` (verified in-container
2026-10-05: precise-memory only swaps the GPU overhead term,
expert_offload_mem only sets the host bound; they rejoin only inside
`place_experts`, and the plan rewrites `expert_offload_mem` back to the cap
the plan met, so plan and allocation cannot drift).
- **Check whether the stack supports the model natively before porting an
overlay.** The image already registered `Glm5NextForConditionalGeneration`,
`Qwen4ExpForCausalLM` and both MTP variants — chasing four missing symbols
through two image tags to bring an overlay up was entirely wasted work.
The checkpoint this recipe was verified on is
`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (gated; 172.8 GiB, 131 shards,
`Qwen4ExpForConditionalGeneration`, 48 layers, 512 experts top-10, multimodal,
native 262,144) — an [[frank-archon]] era payload.
## Compile-cache persistence (survives `podman rm -f`)
Serve scripts that recreate the container discard the writable layer, so
every boot re-pays compile/JIT (frank-archon: `torch.compile took 181.20 s`
cold, of which 173 s was one graph).
1. **Prove it and locate it:** fresh container vs live
(`podman run --rm --entrypoint bash IMG -c "ls -la /root/.cache"` vs
`podman exec LIVE bash -c "du -sb /root/.cache/*; find /root/.cache -type
f | wc -l"`). vLLM cache root: `VLLM_CACHE_ROOT` (default
`/root/.cache/vllm`, `torch_compile_cache/`); a HIP runtime adds
`/root/.cache/comgr`.
2. **Mount the PARENT, not the leaf** — one bind on `/root/.cache` persists
both trees, safe whenever the image ships no such directory (step 1
proves it; otherwise you would be hiding baked state). Wire as an opt-in
tunable: `mkdir -p "$CACHE_DIR"` BEFORE the mount, then
`${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}`. Host path outside the
checkout.
3. **Verify with two consecutive boots** — cold `Compiling a graph … takes
N s`, warm `Directly load the compiled graph(s) for compile range (…)
from the cache` with `torch.compile took` in single-digit seconds.
4. Traps: the cache key includes the compile RANGE — an MNBT change costs
ONE recompile, not a permanent tax; do not claim a proportional boot cut
(time to `/health` is dominated by weight loading: 490 s warm vs 188 s
cold compile); stale entries are misses, not wrong answers (version +
config hashed); if the serve script is seam-gated, assert the mkdir and
the exact mount target.
## Plan-line verification checklist (after any fork deploy)
The journal one-liner that reads the plan lines (digit-anchored `grep -oE`,
because plain grep truncates those columns to `17.`) and the post-deploy
verification order — plan vs captured graph memory, `GPU KV cache size` vs the
ask, per-card `rocm-smi` high-water, `NRestarts` = 0, `VLLM_CUDAGRAPH_*` present
in the container env, then a real completion from a PEER because
`curl localhost:7770/health` inside an ssh session on the host can return `000`
and look like an outage — are step 3 of
[[RDNA4 MoE Expert Offload VRAM Budget]] §7. Run that; do not re-derive it
here.
## Related
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan law this harness tunes
- [[vLLM Boot Forensics and Config Interrogation]] — journal forensics for these boots
- [[LLM Bench Windows and Measurement]] — window discipline and bash laws
- [[Model Swap and Rollback]] — where the fork sits in the swap history
- [[Remote Job Runner]] — the job.sh harness referenced above
serverAppConfigs-docs/LLM/VLLM Index.md
-104
View File
@@ -1,104 +0,0 @@
---
type: reference
description: "This index contains all documentation for vLLM deployment on Dankserver and Dankclaw."
tags:
- llm
- vllm
timestamp: 2026-08-28
---
# vLLM Index
> **RETIRED (2026-08-18).** The vLLM service on Dankclaw was replaced by the CIRU ROCmFPX llama.cpp runtime (v3). These pages are historical reference. See [[frank-dragoon]].
## Documentation Overview
This index contains all documentation for vLLM deployment on Dankserver and Dankclaw.
## Hardware Guides
- [[Dankserver System Specs]] — NVIDIA GPU server specifications
- [[Dankclaw System Specs]] — AMD GPU server specifications
## vLLM Architecture
- [[VLLM Overview]] — Complete vLLM documentation including installation, CLI, API, and key arguments
- [[RAM CPU Offloading in VLLM]] — CPU RAM offloading mechanisms, swap space, and memory configuration
- [[Hybrid CPU GPU Processing Dankserver]] — NVIDIA GPU with system RAM hybrid processing
- [[Shared Memory Architecture Dankclaw]] — AMD ROCm unified memory and APU configuration
## Model Deployment Guides
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B FP8 deployment on NVIDIA GPU
- [[Running Qwen3.6 27B FP8 on Dankclaw]] — Qwen3.6-27B FP8 deployment on AMD GPU
## Quick Reference
### Installation Commands
```
# NVIDIA GPU
uv pip install vllm --torch-backend=auto
# AMD ROCm
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
# CPU
uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly/cpu/
```
### Server Start Commands
```
# Basic server
vllm serve Qwen/Qwen2.5-1.5B-Instruct
# With tensor parallelism
vllm serve meta-llama/Llama-3.1-70B-Instruct --tensor-parallel-size 4
# With quantization
vllm serve Qwen/Qwen2.5-1.5B-Instruct --quantization fp8
# With GPU memory control
vllm serve Qwen/Qwen2.5-1.5B-Instruct --gpu-memory-utilization 0.8 --kv-offloading-size 8 --kv-offloading-backend native
```
### Common API Endpoints
| Endpoint | Method | Description |
|----------|--------|-------------|
| /v1/models | GET | List available models |
| /v1/completions | POST | Text completion |
| /v1/chat/completions | POST | Chat completion |
| /health | GET | Health check |
| /metrics | GET | Prometheus metrics |
### Key Parameters
| Parameter | Description | Typical Value |
|-----------|-------------|---------------|
| tensor_parallel_size | Number of GPUs for model sharding | 1-8 |
| gpu_memory_utilization | Fraction of GPU memory for KV cache (default 0.92) | 0.7-0.95 |
| swap_space | Retired; live knob pair in [[RAM CPU Offloading in VLLM]] | n/a |
| max_model_len | Maximum context length | auto |
| max_num_batched_tokens | Max tokens per batch | 2048-8192 |
| max_num_seqs | Max sequences per batch | 128-512 |
### Quantization Methods
| Method | GPU Support | Description |
|--------|-------------|-------------|
| FP8 | Hopper, Ada (native); Ampere W8A16 via Marlin; RDNA via emulation | FP8 E4M3 inference |
| INT8 | Most GPUs | Weight-only INT8 |
| GPTQ | Most GPUs (NVIDIA; limited on ROCm) | Post-training GPTQ |
| AWQ | Turing+ (NVIDIA) | Activation-aware weight quantization |
| GGUF | CPU, GPU | General GPU unification format |
## See Also
- [[VLLM Overview]]
- [[RAM CPU Offloading in VLLM]]
- [[Hybrid CPU GPU Processing Dankserver]]
- [[Shared Memory Architecture Dankclaw]]
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
- [[Running Qwen3.6 27B FP8 on Dankclaw]]
serverAppConfigs-docs/LLM/VLLM Overview.md
-287
View File
@@ -1,287 +0,0 @@
---
type: reference
description: "vLLM is a fast and easy library for large language model inference and serving."
tags:
- llm
- vllm
timestamp: 2026-08-28
---
# vLLM Overview
## What Is vLLM
vLLM is a fast and easy library for large language model inference and serving. You can use vLLM to run models on NVIDIA GPUs, AMD GPUs, Intel GPUs, and CPUs. vLLM supports over 200 model architectures from Hugging Face.
vLLM is fast with these features:
- PagedAttention for efficient key and value memory management
- Continuous batching of incoming requests
- Chunked prefill and prefix caching
- FP8, INT8, INT4, GPTQ, AWQ, and GGUF quantization
- Tensor, pipeline, data, expert, and context parallelism
- Speculative decoding with n-gram, suffix, EAGLE/EAGLE3, DFlash, DSpark, and MTP
- Automatic kernel generation with torch.compile
- Disaggregated prefill, decode, and encode
vLLM is flexible with these features:
- Seamless integration with Hugging Face models
- High-throughput serving with parallel sampling and beam search
- Streaming outputs
- Structured outputs with xgrammar or guidance
- Tool calling and reasoning parsers
- OpenAI-compatible API server
- Anthropic Messages API and gRPC support
- Efficient multi-LoRA support
## Installation
You can install vLLM with uv (recommended) or pip:
```
uv pip install vllm
```
Or with pip for NVIDIA GPUs:
```
pip install vllm --extra-index-url https://download.pytorch.org/whl/cu129
```
For AMD GPUs with ROCm:
```
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
```
For CPU:
```
uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly/cpu/
```
## Quick Start
### Offline Batched Inference
```python
from vllm import LLM, SamplingParams
llm = LLM(model="Qwen/Qwen2.5-1.5B-Instruct")
prompts = [
"Hello, my name is",
"The capital of France is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
generated_text = output.outputs[0].text
print(generated_text)
```
### Online Serving
Start the OpenAI-compatible API server:
```
vllm serve Qwen/Qwen2.5-1.5B-Instruct
```
Query the server:
```
curl http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-1.5B-Instruct",
"prompt": "San Francisco is a",
"max_tokens": 7,
"temperature": 0
}'
```
Chat completions:
```
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-1.5B-Instruct",
"messages": [
{"role": "user", "content": "Hello!"}
]
}'
```
## Key Arguments
### Model Arguments
| Argument | Default | Description |
|----------|---------|-------------|
| `model` | required | Model name or path |
| `tokenizer` | same as model | Tokenizer name or path |
| `tokenizer_mode` | auto | Tokenizer mode |
| `trust_remote_code` | false | Trust remote code from Hugging Face |
| `dtype` | auto | Data type (auto, float16, bfloat16, float32) |
| `max_model_len` | auto | Maximum context length |
| `quantization` | null | Quantization method (awq, gptq, fp8, compressed-tensors) |
| `load_format` | auto | Weight loading format |
### GPU Arguments
| Argument | Default | Description |
|----------|---------|-------------|
| `tensor_parallel_size` | 1 | Number of GPUs for tensor parallelism |
| `pipeline_parallel_size` | 1 | Number of GPUs for pipeline parallelism |
| `gpu_memory_utilization` | 0.92 | Fraction of GPU memory for KV cache |
| `swap_space` | — | retired; the live CPU-KV-offload knob pair is in [[RAM CPU Offloading in VLLM]] |
| `enforce_eager` | false | Use eager mode (no CUDA graphs) |
| `max_num_batched_tokens` | 2048 | Max tokens per batch |
| `max_num_seqs` | 128 | Max sequences per batch |
| `block_size` | 16 | Token block size |
| `num_gpu_blocks_override` | null | Override number of GPU blocks |
| `max_model_len_per_seq` | null | Max length per sequence |
### Optimization Arguments
| Argument | Default | Description |
|----------|---------|-------------|
| `disable_log_stats` | false | Disable logging |
| `disable_log_requests` | false | Disable request logging |
| `max_log_len` | null | Max log length |
| `disable_custom_all_reduce` | false | Disable custom all-reduce |
| `tokenizer_pool_size` | 0 | Tokenizer pool size |
| `tokenizer_pool_type` | none | Tokenizer pool type |
| `enable_chunked_prefill` | auto | Enable chunked prefill |
| `max_num_batched_tokens` | 2048 | Max tokens per batch |
| `attention_backend` | auto | Attention backend (flash_attn, flashinfer) |
### Memory Arguments
| Argument | Default | Description |
|----------|---------|-------------|
| `gpu_memory_utilization` | 0.92 | Fraction of GPU memory for KV cache |
| `swap_space` | — | retired; the live CPU-KV-offload knob pair is in [[RAM CPU Offloading in VLLM]] |
| `kv_cache_dtype` | auto | KV cache data type (auto, fp8, fp8_e5m2) |
| `max_model_len` | auto | Maximum context length |
| `num_gpu_blocks` | auto | Number of GPU blocks |
| `num_cpu_blocks` | auto | Number of CPU blocks |
### Device Arguments
| Argument | Default | Description |
|----------|---------|-------------|
| `device` | auto | Device type (cuda, rocm, xpu, cpu, metal) |
| `distributed_executor_backend` | auto | Distributed executor backend |
| `huggingface_tokenizer_kwargs` | null | Hugging Face tokenizer kwargs |
## Architecture
vLLM uses a multi-process architecture:
- API server process handles HTTP requests, tokenization, and input processing
- Engine core process runs the scheduler and coordinates GPU workers
- One GPU worker process per GPU executes model forward passes
vLLM uses PagedAttention for memory management. PagedAttention treats KV cache as a set of memory pages. This allows efficient memory allocation and reduces fragmentation.
vLLM supports multiple attention backends:
- FlashAttention for NVIDIA GPUs
- FlashInfer for NVIDIA GPUs
- Triton attention for AMD ROCm
- ROCm attention for AMD ROCm
## Supported Models
vLLM supports over 200 model architectures including:
- Decoder-only LLMs (Llama, Qwen, Gemma)
- Mixture-of-Expert LLMs (Mixtral, DeepSeek-V3, Qwen-MoE)
- Hybrid attention and state-space models (Mamba, Qwen3.5)
- Multi-modal models (LLaVA, Qwen-VL, Pixtral)
- Embedding and retrieval models (E5-Mistral, GTE, ColBERT)
- Reward and classification models (Qwen-Math)
## Quantization
vLLM supports these quantization methods:
- FP8 for Hopper and Ada GPUs (native). Ampere runs FP8 W8A16 via Marlin; FP8 W8A8 computes via fallback
- MXFP8/MXFP4 for Hopper GPUs
- NVFP4 for Hopper and Blackwell GPUs
- INT8 for most GPUs
- INT4 for most GPUs
- GPTQ for most GPUs
- AWQ for Turing and newer GPUs (NVIDIA; native support varies on ROCm)
- GGUF for CPU and GPU
- Compressed-tensors for custom quantization
- ModelOpt for NVIDIA Model Optimizer
- TorchAO for PyTorch AO
AMD RDNA 3 / RDNA 3.5 (gfx11, Strix Halo) run FP8 via software emulation, not native tensor cores. AWQ and GPTQ support in vLLM on AMD ROCm is limited; use GGUF or AMD Quark.
## Parallelism
vLLM supports multiple parallelism strategies:
- Tensor Parallelism (TP) shards model parameters across GPUs
- Pipeline Parallelism (PP) distributes model layers across GPUs
- Data Parallelism (DP) replicates the model across GPU sets
- Expert Parallelism (EP) distributes expert networks in MoE models
- Context Parallelism (CP) splits context across GPUs
## Chunked Prefill — the flag surface, and what it actually does
**YES, the runtime exposes chunked-prefill options, and the running shape
already uses them.** The tcclaviger fork is a vLLM fork, so the standard
scheduler flags are present in the shipped image (verified by reading the
`vllm.engine.arg_utils` and `vllm.config.scheduler` source in the live
container `frank-archon-qwen`, not from a stub):
| Flag | Default | What it controls |
|---|---|---|
| `--enable-chunked-prefill` | `True` in `SchedulerConfig`; the CLI arg is an auto-flag (`"default": None`) | whether prefill may be split into chunks |
| `--max-num-batched-tokens` (MNBT) | 2048 (`DEFAULT_MAX_NUM_BATCHED_TOKENS`) | tokens processed per prefill iteration |
| `--prefill-schedule-interval` | 0 | gap between prefill iterations |
The running shape's argv carries `--max-num-batched-tokens 2048` alongside
`--max-model-len 200704`, so chunked prefill is **ON and already active**: a
200704-token prompt is prefilled in ~98 iterations of 2048 tokens, not in one
shot.
**It is a TTFT lever, not a throughput lever.** MEASURED 2026-09-29 on the
ad hoc qwen-fp8 container (the same fork, same payload, same TP=2):
- MNBT 8192 vs the default 2048: **neutral** on decode at every depth.
- Small values re-chunk prefill only. They hurt TTFT (each chunk is its own
iteration, so the first token waits on more iterations) and they never touch
decode: a decode step is `max_num_seqs x (spec+1)` tokens, independent of
MNBT.
**There is no `--ubatch-size` equivalent in vLLM.** `--ubatch-size` is a
llama.cpp compute-buffer concept (`--batch-size`/`--ubatch-size`); the tcclaviger
fork has no ubatch knob at all. The llama.cpp compute-buffer pair that frank-zealot
was doubled on (2026-10-05, +4.6 % prefill at 8,928 tokens for 616 MiB) has no
counterpart here, and the two backends should not be reasoned about by analogy.
**MNBT is not a lever on this shape either.** The deployed 2048 is the fork's
default, so a change is a deliberate departure from the shipped value, not a
tune. The one thing MNBT does cost: every change re-enters the compile RANGE
path, so `FRANK_ARCHON_CACHE_DIR` pays one recompilation.
## See Also
- [[VLLM Index]]
- [[RAM CPU Offloading in VLLM]]
- [[Hybrid CPU GPU Processing Dankserver]]
- [[Shared Memory Architecture Dankclaw]]
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
- [[Running Qwen3.6 27B FP8 on Dankclaw]]
serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md
+8 -8
View File
@@ -1,7 +1,7 @@
---
type: research
type: reference
title: Dankclaw IOMMU boot line — decision record (amd_iommu=off 2026-09-11 → iommu=pt 2026-10-18)
description: Dankclaw boots iommu=pt since 2026-10-18 — identity DMA with the IOMMU armed, which restored the XDNA2 NPU at a measured prefill cost of nothing (+2.0% on byte-identical traffic). Records the 2026-09-11 amd_iommu=off era that bought a vendor-claimed 13-16% figure never measured here, what that cost, the AMD PerfOpt research (Linux 7.4), the tooling that owns the boot line, and the revert path.
description: "Why Dankclaw boots with identity DMA, what the earlier off-era bought and cost, and which tooling owns the boot line."
tags:
- dankclaw
- iommu
@@ -19,7 +19,7 @@ timestamp: 2026-09-11
**Current state (since 2026-10-18 01:24): Dankclaw boots `iommu=pt`** — identity
DMA, IOMMU armed, XDNA2 NPU bound and usable. Everything below is the record of
the 2026-09-11 decision that booted it with **`amd_iommu=off`** on a vendor-claimed
13–16 % prefill figure, what that cost, and the measured flip back: see
13–16 % prefill figure, what that cost, and the measured flip back:
§ *2026-10-18 execution* at the bottom for the numbers and the revert path.
## What changed
@@ -54,7 +54,7 @@ takes the NPU with it.
1. **The NPU endpoint dies.** With IOMMU off there is no PASID/SVA state and
`/dev/accel/accel0` cannot be opened — `err=-19`, the exact failure
documented in [[frank-npu]]. Audio input left the fleet with it; image input
documented in . Audio input left the fleet with it; image input
survives on `frank-dragoon` through halogen's vision tower.
2. **Posture change machine-wide.** Dankclaw also serves Caddy (the TLS edge
and the Anubis/Authentik guard chain), Authentik, Pi-hole, Gitea, and the
@@ -135,7 +135,7 @@ mode**". So `pt` is most of the recovery and PerfOpt is the last part of it.
**The NPU is compatible with identity mode.** `amdxdna` binds through
`iommu_sva_bind_device()` + PASID + GCR3 into its own `IOPT_TYPE_SVA` domain,
**independent of the default domain type** (`drivers/accel/amdxdna/aie2_pci.c`),
**independent of the default domain type** (`drivers/accel/amdxdna/aie2_pci.c`)
and PerfOpt is not on that path at all. It only needs the IOMMU *enabled* — that
is the whole `err=-19` story. Kernel pick notes: the Ubuntu 6.17 `-95` SVA
failures were a GCR3 regression fixed before our kernel, but **6.18–6.18.7
@@ -204,7 +204,7 @@ depth 2, prompt cache ON, no clamp line — and Gitea, the Caddy edge and Grafan
answered 200. (Later the same day the pool DOUBLING to 401,408 positions for
slot parity was COMMITTED (`2b12716`) and **deployed the same evening**: the
boot now reserves `401408 positions (about 11.0 GiB)` and the engine-side total
is `79.6 GiB in all` (62.9 weights + 11.0 KV + 5.7 working),
is `79.6 GiB in all` (62.9 weights + 11.0 KV + 5.7 working)
see [[frank-dragoon]].)
**The NPU came back with the flip.** `amdxdna` bound to `0000:c6:00.1` (Strix/
@@ -212,7 +212,7 @@ Krackan/Strix Halo NPU rev 11), firmware `amdnpu/17f0_11/npu_7.sbin`, `Initializ
amdxdna_accel_driver 0.7.0 … on minor 0`, IOMMU group 26 `identity`, PASID cap width
10 (`Exec- Priv-`). `/dev/accel/accel0` is `crw-rw---- root:render 261,0` and `dank`
is in `render`, so a quadlet needs `AddDevice=/dev/accel/accel0` +
`GroupAdd=keep-groups`. **No NPU endpoint is deployed** — [[frank-npu]] stays the
`GroupAdd=keep-groups`. **No NPU endpoint is deployed** — stays the
history of the retired endpoint; what changed is only its precondition.
**Deliberately deferred:** the `pp_dpm_fclk` pin (level 7, 2000 MHz) from the issue's
@@ -228,7 +228,7 @@ lives in `~/.local/bin`.
## Related
- [[frank-npu]] — the endpoint this retired, including the err=-19 root cause
- the endpoint this retired, including the err=-19 root cause
- [[frank-dragoon]] — the runtime bought by the prefill figure
- [[frank-dragoon]] — the runtime it replaced (rollback target)
serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md
-227
View File
@@ -1,227 +0,0 @@
---
type: research
title: Frontier MoE on Dankserver — what fits, on which runtime
description: VERIFIED 2026-09-28 by measurement and source reading. Compares DeepSeek-V4.1-Flash, MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash against the 2x R9700 (gfx1201) / 246 GiB box, and assesses the Affinity runtime's portability off DeepSeek-V4-Flash. Records the capacity arithmetic that decides the answer before any model is loaded.
tags:
- dankserver
- r9700
- gfx1201
- moe
- deepseek-v4.1
- deepseek-v4
- mimo
- glm
- affinity
- lucebox
- feasibility
timestamp: 2026-09-28
---
# Frontier MoE on Dankserver — what fits, on which runtime
> **Question this page answers.** Given Dankserver's hardware, which of
> DeepSeek-V4.1-Flash, MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash runs best, on
> which runtime, and can Affinity / vllm-radiance / FreeToken-AMD be modified
> to run them fast?
>
> **Answer in one line.** V4.1-Flash does not fit this box at any
> configuration; the engine that does run it (Lucebox) measures
> 0.8–9.1 tok/s on R9700 because decode is bound by PCIe expert streaming
> rather than by the GPUs. The 284B **V4-Flash** on **Affinity** is the only
> shape that clears 35 tok/s, at 110.
Everything below was measured or read from source on 2026-09-28. Claims
carried over from the engine authors are marked as theirs.
## Hardware (measured live, 2026-09-28)
| Property | Value |
|---|---|
| GPUs | 2x AMD Radeon AI PRO R9700 (Navi 48, **gfx1201**), 32,624 MiB each = **63.7 GiB** total |
| Interconnect | PCIe 5.0 **x8** each, CPU root ports `00:01.1` / `00:01.3`, ~31.5 GB/s per card |
| PCIe atomics | `CAP_EXP+28.w = 0x1440` (ReqEn+) on both endpoints — RCCL collectives are legal |
| CPU | Ryzen 7 7700X — **8 cores / 16 threads** (small; CPU expert compute is expensive) |
| RAM | **246 GiB** + 63 GiB swap |
| Disk | 1.8 TB NVMe. `/mnt/entropy` is a separate 91 TB ext4 at **100%** — do not plan payload there |
The knowledge base was wrong about RAM: [[Dankserver System Specs]] said
60 GiB until 2026-09-28 and has been corrected. The Lucebox runbook's 246 GiB
was right.
## The three candidates
| Model | Total / active | Routed experts | Attention & KV | Working gfx1201 runtime |
|---|---|---|---|---|
| **DeepSeek-V4.1-Flash** | 552B / 8B prefill, 16B decode | 384 top-6, 40 layers, `moe_intermediate_size` 2304, hidden 5120 | CED 20 enc + 20 dec; CSA2 Full/Reindex/Reuse; **FP4 main KV, 890 B/token**; Engram 196B at layers 1 & 14 | Lucebox `feat/ds41-engine` only (unmerged) |
| **MiMo-V2.6-Flash-MOPD** | 309B / 15B | 256 top-8, no shared expert, 48 layers (39 SWA / 9 GA) | hybrid SWA (window 128) + GA; 5-layer MTP | vLLM / SGLang, NVIDIA-first. No AMD path |
| **GLM-5.3-Flash** | 320B / 18B | MoE | KDA (linear) + DSA (sparse) hybrid; mHC | **tcclaviger vLLM fork — plausible native path (2026-09-29 source reading: `glm5next` registered with an `amd/` backend dir + DSA indexer + mHC path; r4d.so carries native mxfp4/dsfp4 MoE entries). Gate: a compressed-tensors MXFP4 payload (~165 GiB, none exists publicly) + a HIP boot probe.** llama.cpp GGUF (community R9700 guide) remains the verified-but-slower path |
## Sizing — the arithmetic that decides it
| Quantity | Value | Note |
|---|---:|---|
| DS4.1 official checkpoint | **475.27 GiB** | 48 shards, summed from the HF API |
| &nbsp;&nbsp;of which Engram | **189.13 GiB** | exactly two 94.56 GiB shards — the layer 1 and 14 lookup tables |
| DS4.1 routed-expert params | 543.6 B | 40 x 384 x 3 x 5120 x 2304 |
| &nbsp;&nbsp;shipped as | 269 GiB (fp4) | DeepSeek's own `expert_dtype: fp4` |
| Lucebox `ROCMFP23.gguf` | **377.51 GiB** | 179 GiB experts (2.5/3.5 bpw) + 189 GiB Engram + ~9.5 GiB dense |
| V4-Flash official checkpoint | 148.67 GiB | **3.2x smaller than V4.1** |
| Affinity container for V4-Flash | 101.1 GiB | 92.9 GiB experts @ 2.875 bpw + 6.2 dense + 2.0 embed |
The Engram tables are **never mapped** by Lucebox, so they do not evict the
expert page cache — that is the one piece of good news in the 377 GiB file.
### The decode wall
A V4.1 token selects `6 x 40 = 240` routed experts at ~11 MB each. At the
residency the hot stack can hold (37 of 384 per layer, 9.6%), **~2.4 GB of
expert weights cross PCIe per token**. PCIe 5.0 x8 does ~55 GB/s real:
```
2.4 GB / 55 GB/s = 44 ms -> ~23 tok/s absolute ceiling
```
That is **below the 35 tok/s target before any attention, Engram or dense work
is counted.** Raising residency is the only lever, and VRAM is what bounds it:
63.7 GiB / 11 MB = 5,800 of 15,360 experts = 37.8% maximum, and that assumes
VRAM holds nothing else.
The engine author measured the same wall on an R9700 with 61 GB of RAM
(DS41.md, "Measured"): 0.8 tok/s cold / 1.8 tok/s warm at a 6 GB hot budget,
0.9 / 1.9 at 16 GB, and 9.1 tok/s for the best tuned three-tier case. They
name the cause: *"90 to 96% of the 240 routed experts a token selects are still
cold, and decode moves roughly 2.4 GB of expert weights per token through the
synchronous stream path. That, not the GPU, bounds decode today."*
## Affinity — a DeepSeek-V4-Flash engine, and what a V4.1 port costs
[Affinity](https://codeberg.org/StillDeadcode/affinity) is *"an inference
engine for DeepSeek-V4-Flash on one or two RDNA4 GPUs"*, C++/HIP, no framework.
**It builds clean on this host**: CMake 3.24+, `hipcc` 23, `HIP enabled:
gfx1201`, zero errors, producing `affinity`, `aff-quantize`, `aff-info` and
`aff-gpucheck`. Its README claims **~110 tok/s on two R9700s** against 8 on
one. That is 3x Frank's 35 tok/s goal, for the 284B model.
The 110 works because the 101 GiB container leaves the cold expert set in a
**pinned host pool in RAM** — the SSD tier that Affinity calls "a cliff" is
never reached. V4.1 would need 182 GiB of experts at the same 2.875 bpw, plus
189 GiB of Engram, against the same 246 GiB of RAM.
### What a V4.1 port would reuse
The config parser is **fully data-driven** — every dimension is read from
`config.json`, so nothing is compiled in at the config layer. And the expert
tensor naming is *identical* between the two checkpoints:
```
layers.N.ffn.experts.E.w1.weight (11,008 in V4 = 43 x 256)
layers.N.ffn.experts.E.w1.weight (15,360 in V4.1 = 40 x 384)
```
So the quantiser, the `.aff` container, the expert slab, the host pool and the
heat/placement engine — **the part that produces the 110 tok/s** — all carry
over. Hyper-connections are the same family (`hc_mult 4`, Sinkhorn 20).
### What a V4.1 port would have to write
Verified against both checkpoints' real `model.safetensors.index.json` on
2026-09-28 — not inferred from the config prose.
**Tensor presence.** `aff-quantize` requires 32 per-layer tensors. V4.1
supplies 26 plus every global (`embed`, `head`, `norm`, `hc_head_*`) and the
whole expert set. Six are missing:
| Missing in V4.1 | Replaced in V4.1 by |
|---|---|
| `ffn.gate.tid2eid` | the Engram tables |
| `attn.compressor.ape` | nothing — ratio-2 is *pooled, no APE* |
| `attn.indexer.compressor.{wkv,wgate,ape,norm}.weight` | `attn.indexer.wk` + `attn.indexer.k_norm` |
**Config keys.** Of the 35 Affinity reads, V4.1 supplies **34**. The one it
lacks is the dangerous one:
| Gap | Detail |
|---|---|
| **Engram — fails silently** | `num_hash_layers` (V4: 3) is absent, so `cfg_int` returns its default and `model.h:36`'s `n_hash_layer = 3` survives. It would route layers 0–2 through a table that does not exist: fluent output, wrong numbers, no error. This is the only silent failure; the other four gaps fail loudly. |
| **Ratio vocabulary is disjoint** | V4 uses `{0, 4, 128}`; V4.1 uses `{0, 1, 2}` — **no overlap in the non-zero case**. Affinity computes `coff = (ratio == 4) ? 2 : 1`, so V4.1's 18 ratio-2 layers get the ratio-1 width; `comp_ape` is requested 38 times for a tensor that does not exist; and the `ratio == 4` indexer branch fires on **0** of the 8 layers that need one (`index_source_layer_ids` = 2, 8, 14, 20, 24, 28, 32, 36) — those 8 would run dense. |
| **CED** | A 20-layer encoder feeding a 20-layer decoder whose global KV is projected from the final encoder hidden state, through an `aligner`. Affinity's `model.cpp` is a flat layer stack. |
| **DSpark** | V4.1's draft carries **its own** 128-expert MoE at top-3 (`dspark_n_routed_experts`, `dspark_num_experts_per_tok`), 3 stages rather than 1. Affinity's draft shares the target's slab and routing. |
| Vision | V4.1 is multimodal; Affinity has no vision path. Text-only is an acceptable loss. |
V4.1 also adds 31 config keys Affinity never reads, including the whole
`engram_*` block, `kv_source_layer_ids`, `index_source_layer_ids` and the
candidate block pre-selection.
**Verdict.** The reusable half is the *right* half — the expert quantiser,
the `.aff` container, the slab, the host pool and the heat/placement engine
are what produce the 110 tok/s. The rest is 6 tensor bindings, one new ratio
vocabulary, the CED aligner, a new draft geometry and Engram: weeks of work
in a third-party codebase, with no cheap way to prove correctness. And it
meets the same capacity wall regardless, because 182 GiB of experts plus
189 GiB of Engram against 246 GiB of RAM is exactly the problem Lucebox has.
Not advisable.
## The other runtimes
| Runtime | gfx1201 | Architectures | Note |
|---|---|---|---|
| **Affinity** | yes, built here | **V4-Flash only** — "the geometry is compiled in; there is no architecture abstraction" | 110 tok/s for 284B. Best engine for this box by an order of magnitude, wrong model |
| **tcclaviger vLLM fork** | **yes — proven 2026-09-29** | qwen4exp measured 37.4-40.6 tok/s decode (see [[frank-archon]]); **`glm5next` registered** (`Glm5NextForCausalLM`/`MTP`, backend-split `amd/`/`common/`/`nvidia/`, `sparse_indexer.py` for DSA); r4d.so has **native mxfp4 + dsfp4 MoE entries** (`has_dsfp4() == True`); W4A16 MoE class (`R4dW4a16MoEExperts`) wired into the same tiered offload; TunableOp silently skipped under offload (needs `CLAV_TUNABLEOP_WITH_OFFLOAD=1`, and then corrupts numerics — do not use) | the ad hoc qwen-fp8 container's runtime; **the old "MXFP4 = emulation/missing kernels" verdict predates this fork and is dead** — the GLM-5.3-Flash gate is now the payload format + a boot probe, not kernels |
| **Lucebox** | yes, image on host | Qwen 3.8, DS4; **V4.1 on the unmerged `feat/ds41-engine` branch** | The only V4.1 path. **Refuses layer split and layer-major prefill for V4.1** |
| **radiance-vLLM** (drwolfen) | yes — served frank-archon 2026-09-27 → 09-30 | welded to `qwen4_exp`; PLE / QSA / packed-HC live inside that model class | 12–20 person-days per architecture |
| **FreeToken-ROCm** | partial | block-FP8 safetensors only (modelopt); known `CpuMoeExecutor` issue under ROCm | cannot load a GGUF |
| **llama.cpp** | yes | GLM-5.3 confirmed by a community R9700 guide; DS-V4.1 CED unverified | no expert LRU; **8 CPU cores** make CPU expert placement the wrong trade |
| **TokenSpeed** | claimed | GLM-5.x incl. a 5.3-Flash recipe | NVIDIA-first, unverified on gfx1201 |
| **KTransformers** | no | GLM-5.3 tutorial exists | built for Intel AMX + NVIDIA |
| **radiance** (StillDeadcode — the standalone C++/HIP engine, NOT the vLLM fork of the same name) | image `stilldeadcode/radiance` prebuilt for gfx1201 | qwen35 dense/MoE at fp8, qwen4exp quantised-at-convert, llama-shaped | see the probe below: our FP8 payload is REFUSED; tiers, MTP and vision native |
### radiance (standalone) — 2026-10-05 probe
`codeberg.org/StillDeadcode/radiance` — Deadcode's from-scratch engine
(v1.0.13): same author as the `vllm-radiance` fork we ran in September
([[frank-medium-radiance-mxfp4]]), an unrelated codebase — C++/HIP, no
Python, kernels/architectures/quantisers as `.so` plugins, flags written
for two 32 GB R9700s, native Qwen3.8-Flash-Next with `--placement
expert_tiered`, MTP and vision.
Probe with the DEPLOYED payload (`rad-convert --plan-only`, which stops
at declare — no GPU touched, no bytes written):
```
E no architecture plugin claims 'qwen4exp' at quantisation 'fp8'.
'qwen4exp' is served at: (unquantised).
```
So the 173 G orcarouter FP8 file cannot enter this engine at all: the
qwen4exp plugin takes bf16/f16/f32/GGUF in and every quantisation happens
at `rad-convert` time by recipe. Not a lock-in to their published quant —
recipes are per-weight-class rules (`rtn`/`gptq`/`awq`/`autoround`/
`paroquant`/`ggml`, grids from fp8 128×128 to i4/w4nl/w5nl codebooks,
custom `--quant` rules go ahead of the file's), and libr4d lists routed
experts-as-fp8 among the encodings it reads — so today's quantization is
reproducible **from a bf16 source**, not from our file. Uncensored-family
bf16 source candidates: `windowsxp811203/Qwen3.8-Flash-Next-Abliterated-GGUF`
(ships the bf16 GGUF) and `mradermacher/Qwen3.8-Flash-Next-Uncensored-GGUF`
(i1 quants only). 622 G free fits source + container. A trial is:
download → `rad-convert` (CPU-side) → stop+mask frank-archon window to
serve. Caveats on record: `--tp-wire wht6` is a lossy int8 all-reduce
wire; the recurrent GDN state is f32 by default (144 MiB/sequence) with
an opt-in f16 that changes generated text; every pool is budgeted
explicitly — no profiling pass, no utilisation fraction.
The two-R9700 three-owner split (primary + secondary + stream) is **not
available on this host**: the retired quadlet recorded that
`LUCE_DS4_MOE_TP_INPROC` is qualified for gfx1151+gfx1201 pairs, *not*
gfx1201+gfx1201. hip:0 takes dense weights and the hot expert stack;
`LUCE_EXPERT_STREAM_DEVICE=1` gives hip:1 the streamed-expert slot cache.
## Measured on Dankserver
<!-- MEASUREMENT -->
## See Also
- [[frank-archon]] — the retired Lucebox frank-archon runbook
- [[Dual-R9700-Cooperation]] — the PCIe/atomics facts behind the board swap
- [[Dankserver System Specs]] — the host (RAM corrected 2026-09-28)
- [[frank-archon]] — the ACTIVE frank-archon
serverAppConfigs-docs/LLM/dankserver-vram-growth.md
-144
View File
@@ -1,144 +0,0 @@
---
type: reference
title: Dankserver VRAM and RAM Growth Under Load
description: Investigation of the recurring cuBLAS crash on frank-zealot. What grows, what does not, and what to do next.
tags:
- dankserver
- llamacpp
- vram
- ram
- ctx-checkpoints
- crash
timestamp: 2026-09-03
---
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
> See [[frank-redo-2026-08-17]] and [[deployment]] for the active configuration.
> The `frank-zealot` below is the 2026-08 service: llama.cpp on the RTX 3090
> **while that card sat in Dankserver**. Today's [[frank-zealot]] is a different host
> (Dankclaw), payload and runtime build, though the same card class - this crash
> signature is not a live fault.
# Dankserver VRAM and RAM Growth Under Load
This page records the investigation of the recurring cuBLAS crash on
`frank-zealot` (llama.cpp, RTX 3090 24 GiB). The service aborts
with `CUDA error: an unsupported value or parameter was passed to the
function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541` during
`update_slots` -> `decode` -> `process_ubatch`. It crashed on 2026-08-13
at 19:45 and 19:55 and on 2026-08-14 at 12:00:43.
## What Was Ruled Out
### ctx-checkpoints do not grow VRAM
`--ctx-checkpoints 4` stores context checkpoints in host RAM. The log
records every checkpoint at exactly 62.813 MiB. The size never grows.
The reasoning:
- `common_prompt_checkpoint` (common/common.h) stores the serialized KV
state in `std::vector<uint8_t>` fields. These live on the host heap.
- `update_tgt()` and `update_dft()` call `llama_state_seq_get_data_ext`
with `LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY`. The serialized state moves
from VRAM to RAM for the snapshot and back on restore.
- With 4 slots and `--ctx-checkpoints 4`, the maximum checkpoint RAM is
16 x 62.813 MiB, about 1.0 GiB. That is fixed and small.
- `data_spec` (spec-decode state) is empty for ngram-mod. The ngram-mod
implementation does not override `get_state()`.
### ngram-mod does not grow VRAM
ngram-mod allocates one fixed 64 MiB CPU hash table
(`common_ngram_mod::entries`, 16M int32 entries). It never resizes. The
matcher keeps about 12 bytes per sequence. Draft generation runs on the
CPU. The speculative path allocates zero GPU memory.
### The CUDA graph cache has eviction in this build
`ggml_backend_cuda_context::cuda_graph()` sweeps every 5 seconds and
evicts graphs unused for 10 seconds. Upstream issue #25082 describes an
unbounded graph cache, but build b10108-13-g555881ebc includes the
eviction sweep. This build does not exhibit that leak.
## What Grows
### RAM growth is suspected, not yet confirmed
systemd reported MemoryPeak 46.9 GiB for the crashed process and 45.9
GiB for the replacement. RSS is flat at 32.3 GiB, which matches the
documented idle baseline of about 32 GiB. The difference is file-backed
page cache from the `--no-mmap` model read; cgroup memory.current counts
file pages attributed to the process. That gap is not proof of a leak by
itself.
The upstream pattern to watch is issue #26216: the prompt cache and
checkpoint accumulation exceed the `--cache-ram` cap and do not plateau.
The environment file does not set `--cache-ram`, so the llama.cpp
default of 8192 MiB applies. The monitor logs RSS, which separates
anonymous memory from file cache. If RSS climbs under load while VRAM is
flat, the RAM-side accumulation is real.
### VRAM suspects under load
VRAM is flat at idle: 21,268 MiB across every monitor sample. The crash
correlates with load density and slot context size (143K tokens on
slot 2), not with wall-clock time. The two mechanisms worth testing:
1. Flash-attention F16 K conversion workspace. The prefill kernel on
Ampere is `MMA_F16`. It converts the quantized K cache (q8_0) to F16
for the attention math. The workspace size scales with the attended
context length (`ggml_nelements(K) x 2`). It is allocated into the
graph buffer via `ggml_cuda_flash_attn_ext_get_alloc_size`.
2. The CUDA VMM pool grows monotonically. `ggml_cuda_pool_vmm::alloc`
only reserves more physical memory; `free` only releases the
reservation slot. The pool never shrinks during the process
lifetime. Any new peak allocation permanently expands VRAM usage.
Both match the observed pattern: growth under prefill bursts at deep
context, flat at idle, crash when a burst exceeds the ~2.8 GiB headroom.
## Monitoring
The monitor runs on Dankserver:
```
/home/dank/vram-monitor.sh
/home/dank/vram-log.csv
```
It samples GPU VRAM and process RAM every 10 seconds and appends a CSV
row. The columns are timestamp, uptime, GPU VRAM MiB, GPU utilization,
process RSS, process VSZ, and systemd MemoryPeak.
## Resolution: dual-f16 KV Cache (2026-08-14)
The fix chose the first mechanism over the second. Both K and V moved to f16 on 2026-08-14. When K is already f16, the flash-attention MMA_F16 prefill kernel needs no F16 conversion copy, so the context-proportional workspace disappears. The pool becomes static: 20,480 bytes per cell (10 full-attention layers x 2 kv heads x 256 dims x 2 bytes for K plus 2 bytes for V).
The pool at 800768 cells cost 15,640 MiB and did not fit. The load-time warmup graph (1472x1472 image) failed cudaMalloc by 248 MiB, and the service idled at 23,966 of 24,576 MiB. The context pool was refitted to 700416 cells (4 slots x 175104): KV pool 13,680 MiB, idle 22,110 MiB, about 2.0 GiB free after warmup. Parallel stays 4.
Verified 2026-08-14 after deploy: flash attention enabled, warmup clean, decode about 42-44 t/s, cached prompt tokens working. The vram monitor log shows the restart arc (23,966 -> 5,210 during load -> 22,110 idle).
## Mitigation Options
The documentation for the benchmark ([[frank-zealot]])
prescribes the first option for this crash class. None of these change
the model.
| Option | Effect | Cost |
| ------ | ------ | ---- |
| `GGML_CUDA_FORCE_MMQ=1` | Bypasses the cuBLAS path in the crashing function | Slight decode slowdown |
| `--cache-ram 2048` | Bounds the confirmed RAM growth | Cache misses on prompt reuse |
| `--ctx-checkpoints 2` | Cuts checkpoint RAM and save cost in half | More prefill work after eviction |
| Apply PR #24649 | Clears slot checkpoints before the prompt-cache save | Needs a llama.cpp rebuild |
Upstream references: issue #26216 (prompt cache growth), issue #25082
(graph cache, not applicable to this build), issue #23635 (quantized KV
with speculative decoding), PR #24649 (checkpoint-to-cache duplication),
PR #25649 (slot checkpoints freed on prompt clear, merged), issue #26119
(same cuBLAS signature, confirmed on V100 with speculative decoding).
## See Also
- [[frank-zealot]]
- [[Dankserver System Specs]]
- [[deployment]]
serverAppConfigs-docs/LLM/dealign-ornith-crack.md
-126
View File
@@ -1,126 +0,0 @@
---
type: reference
title: dealign Ornith-1.5-35B-A3B CRACK on Dankserver
description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with ngram-mod speculative decoding; selected expert weights offloaded to CPU with an explicit --override-tensor. frank-zealot.
tags:
- llamacpp
- cuda
- dankserver
- moe
- ornith
- mtp
timestamp: 2026-08-24
---
# dealign Ornith-1.5-35B-A3B CRACK on Dankserver
> **Active 2026-08-25.** `frank-zealot` serves this model. It
> replaced the CRACK Q8_0 file on the same upstream llama.cpp binary (b10121).
> The Q6_K file was downloaded from the model repository and validated with
> `llama-gguf` before installation.
`frank-zealot` (port 7770) serves
`Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` from `dealignai` on Hugging Face. It is
a **MoE** model (256 experts, 8 active) with a hybrid GatedDeltaNet (SSM) +
attention architecture, abliterated (uncensored). The GGUF also contains a
bundled MTP head, but this deployment uses ngram-mod instead.
## Model Files
| File | Size |
|------|------|
| `Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` | 27.2 GiB |
| `mmproj-Ornith-1.5-35B-A3B-f16.gguf` (vision) | 0.84 GiB |
Store the files under `/home/dank/models/Ornith-1.5-35B-A3B-CRACK/`.
## Layer Audit and Expert Offload
The Q6_K file contains 41 transformer blocks and 753 tensors. Each block
contains 630 MiB of expert tensors. Blocks 3, 7, 11, 15, 19, 23, 27, 31,
35, and 39 contain full attention. Block 40 contains the bundled MTP head.
The non-expert tensors total 2.02 GiB. The layer totals are 660.96 MiB for
ordinary SSM blocks, 655.81 MiB for full-attention blocks, and 662.40 MiB for
block 40. The expert portion is 630 MiB in every block.
The selected configuration keeps all non-expert tensors on CUDA. It also
keeps expert tensors from every full-attention block, plus every block from 35
through 40, on CUDA. The unique GPU expert set is
`3,7,11,15,19,23,27,31,35,36,37,38,39,40`. Expert tensors from all other
blocks move to CPU through this explicit override:
```
OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29|30|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
OVERRIDE_BUFT=CPU
```
The 16K benchmark found the hybrid policy fastest among the three tested
placements. A later no-spec/ubatch sweep established that `ubatch=1024` is a
stable low-memory choice. The current service uses ngram-mod only; MTP is
intentionally disabled. The full-attention expert set and final blocks are
kept on CUDA.
To list the buffer type names on the host, run a bogus override:
```
llama-server --model <ANY> -c 64 -ot foo=BADTYPE
```
On Dankserver the available buffer types are `CPU` and `CUDA0`.
## Speculative Decoding
The GGUF contains a bundled MTP head, but this deployment deliberately does
not use it. The current service enables only `ngram-mod`, with a 48-token
minimum, 64-token maximum, and 24-token lookup match:
```
--spec-type ngram-mod
--spec-ngram-mod-n-min 48
--spec-ngram-mod-n-max 64
--spec-ngram-mod-n-match 24
```
In the controlled 8K test, no speculation produced approximately 53.7–53.9
generation tokens per second. MTP was slower for that workload, so it remains
disabled.
## Launch Excerpt
```
llama-server \
-m Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf \
--mmproj mmproj-Ornith-1.5-35B-A3B-f16.gguf \
--chat-template-file chat-template.jinja \
--spec-type ngram-mod \
--spec-ngram-mod-n-min 48 \
--spec-ngram-mod-n-max 64 \
--spec-ngram-mod-n-match 24 \
--override-tensor "blk\\.(0|1|2|3|4|5|6|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \
-ngl all -c 491520 -np 3 \
--ubatch-size 1024 \
--cache-type-k f16 --cache-type-v f16 --no-mmap \
--temp 1.0 --top-p 0.95 --top-k 20
```
## Known Issues
1. The 27.2 GiB Q6_K file and 0.84 GiB vision projector fit the 62 GiB RAM
box with the selected expert offload.
2. The context pool contains 491520 cells. Three slots receive 163840 cells
each, which is 96K below the model context per stream.
3. `--n-cpu-moe` remains supported. This service prefers the explicit tensor
override because the 16K benchmark found the hybrid placement fastest. It
preserves every full-attention expert block and blocks 35–40 on CUDA.
4. MTP is bundled in the GGUF but intentionally disabled; ngram-mod is the
only active speculative method.
4. The service uses the Qwen-Sharp template from
`peculiar-ragdoll/Qwen-Sharp-Chat-Templates` through
`scripts/chat-template.jinja`.
## See Also
- [[deployment]] — deploy and update process
- [[Dankserver System Specs]] — the box
- [[frank-redo-2026-08-17]] — fleet topology history
serverAppConfigs-docs/LLM/deployment.md
-178
View File
@@ -1,178 +0,0 @@
---
type: guide
title: LLM Deployment
description: Scoped deployment and update process for the three inference services (frank-archon, frank-dragoon, frank-zealot).
tags:
- llm
- deployment
- systemd
timestamp: 2026-09-03
---
# Deployment and Update Process
This page describes how to deploy and update the LLM servers on Dankserver and
Dankclaw. The process uses Git as the single source of truth. A change to
SearXNG, Caddy, or another container service does **not** restart inference.
## Fleet
| Target | Host | Service | Port |
|---|---|---|---:|
| Dankserver | `dank@dankserver` | `frank-archon` | 7770 |
| Dankclaw | `dank@dankclaw` | `frank-dragoon` | 7770 |
| Dankclaw | `dank@dankclaw` | `frank-zealot` | 7771 |
## Scoped workflow
```mermaid
graph LR
A[Edit LLM files] --> B[Run tests]
B --> C[Commit main]
C --> D[deploy.sh selects matching service]
D --> E[Target pulls ff-only]
E --> F[Relink one unit]
F --> G[Restart one inference service]
G --> H[Wait for that service health]
```
Run the unified deployer from the repository root:
```bash
./scripts/deploy.sh dankserver --service frank-archon
./scripts/deploy.sh dankclaw --service frank-dragoon
./scripts/deploy.sh dankclaw --dry-run
```
Without `--service`, `scripts/deploy-plan.py` maps changed files to the service
on the selected host. The following files select the corresponding service:
- `scripts/frank-archon.env`, `frank-archon.service`, and
`serve-frank-archon.sh` select `frank-archon` on Dankserver
(`scripts/deploy-plan.py:422-428`). The quadlet-era directory
`dankserver/frank-medium/` left the tree on 2026-09-26 and plans nothing
(`scripts/deploy-plan.py:374-377`).
- `scripts/qwen-sharp-v22.4.0.jinja` selects `frank-dragoon` — a planner rule
left from the llama.cpp era when dragoon loaded it via
`--chat-template-file` (`scripts/deploy-plan.py:415-421`).
- `scripts/frank-dragoon.env`, `frank-dragoon.service`, and
`serve-frank-dragoon.sh` select `frank-dragoon` on Dankclaw.
- `scripts/frank-zealot.env`, `frank-zealot.service`, and
`serve-frank-zealot.sh` select `frank-zealot` on Dankclaw.
- Documentation, tests, deployment tooling, and unrelated container changes
do not select any inference service.
The target-side `scripts/deploy-remote.sh` installs the selected service.
All three LLM services are **root units** whose repo unit file is relinked
into `/etc/systemd/system/` (`relink_unit`), daemon-reloaded, enabled and
restarted (`deploy-remote.sh:1158` `deploy_llm_dragoon`, `:1254`
`deploy_llm_zealot`, `:1315` `deploy_llm_archon`). The dragoon and archon
handlers wait for port 7770 to be released before the replacement starts
(`wait_port_free`; the 2026-09-16/26 cutover lessons). The archon handler
additionally hard-gates on the env file, the digest-pinned image ref, and the
payload before touching the unit (refuse rather than restart-loop). Each
handler then waits on that service's health endpoint —
`http://127.0.0.1:7770/health` for frank-dragoon (budget 1200 s) and
frank-archon (1800 s), `http://127.0.0.1:7771/health` for frank-zealot
(900 s) — and refuses a false green: the deploy is only green when the unit
itself reports a live MainPID behind the health answer
(`deploy-remote.sh:1395-1403`, `:1285`). A failed handler does not advance
its per-service state.
## Unit and artifact paths
The root units point into the stable target checkout:
```text
/home/dank/serverAppConfigs/scripts/frank-archon.service
/home/dank/serverAppConfigs/scripts/frank-dragoon.service
/home/dank/serverAppConfigs/scripts/frank-zealot.service
```
All three services keep their unit, env and serve script as this flat trio
under `scripts/` — there is no LLM Quadlet left (the `frank-medium`
quadlet-era directory left the tree 2026-09-26).
The unit files and serve scripts validate their model/runtime artifacts before
starting. Keep `REMOTE_DIR` stable; moving the checkout requires updating both
the unit absolute paths and the deployment tests in the same change.
## Retired endpoints
`frank-npu` (the FastFlowLM XDNA2 NPU endpoint) was retired on 2026-09-11
with the `amd_iommu=off` decision and is no longer a deployable service — the
planner has no `frank-npu` entry. Its runbook is [[frank-npu]].
## Isolation guarantee
The remote runner has one explicit service dispatch. It does not run a
host-wide LLM loop, `podman-compose up` for unrelated services, or a blanket
systemd restart. Each successful service is recorded under the target user's
state directory outside Git:
```text
~/.local/state/serverAppConfigs/deploy-state/
```
On a later ordinary deployment, a service remains pending only while its
recorded revision differs from the checkout and the planner still finds a
change in that service's inputs.
## Locally built images
`build_compose_image` in `scripts/deploy-remote.sh` builds the images that have
no registry digest — `comfyui-rocm`, `franksdank/loki`, `franksdank/alloy` and
the voice pair `franksdank/frank-tts` / `franksdank/frank-tts-engine` — and
**skips** the build when a `cfhash` label on the installed image matches a hash
it computes from the checkout now. That skip is what keeps the engine's
ten-minute `git clone --recursive` off every re-deploy, so the rule has to be
honest about what the image bakes:
- The hash covers the `Containerfile` **plus the files its `COPY` lines pull
in**, named as `extra=(…)` in each case arm. Hashing the `Containerfile`
alone was wrong in practice: editing `dankclaw/voiceservice/app.py` or
`tts-engine-warm.sh` left the label matching, `podman build` never ran, and
the redeployed container still carried the previous code while `/health`
answered — invisible to every gate. Corrected 2026-10-18;
`tests/frank_tts_engine.bats` drives the function against a stub `podman`
(first deploy builds, unchanged tree does not, a `COPY`'d file rebuilds).
- A base-image assumption is part of the image's identity too. The engine's
runtime stage creates its unprivileged user with `useradd`, not `adduser`,
because the pinned `ubuntu:24.04` base ships no `adduser` — checked by
running the command against the digest on 2026-10-18. That failure mode is a
build that dies at the `RUN` line, so it is loud, unlike the stale-hash one.
Both engines keep serving during a rebuild: `build_compose_image` runs after
`install_quadlet` only for the LLM-style handlers, and the voice deploy gates on
recorded health rather than a fixed sleep — see [[Frank TTS Engine]].
## Manual recovery
If the LLM handler fails after the target pull:
1. Inspect `journalctl -u frank-dragoon -n 200`, `journalctl -u frank-zealot
-n 200` or `journalctl -u frank-archon -n 200` (all three are root units);
frank-dragoon's engine boot ledger is in `/home/dank/logs/frank_dragoon.log`,
frank-archon's in `podman logs frank-archon` — not in the journal.
2. Confirm the model files, the runtime binary and (for frank-archon) the
digest-pinned container image are present and readable by `dank`.
3. Retry only the affected service:
```bash
./scripts/deploy.sh dankserver --service frank-archon
./scripts/deploy.sh dankclaw --service frank-dragoon
./scripts/deploy.sh dankclaw --service frank-zealot
```
4. Verify the matching health endpoint before declaring recovery.
Do not stop the other host's inference service during recovery. The two
Dankclaw endpoints are independent processes on different devices: restarting
frank-dragoon does not disturb frank-zealot, and the reverse holds too — with one
exception worth knowing, that the deployer waits for port 7770 to be released
before it starts the replacement container. For benchmark work, follow the
separate stop/benchmark/restart procedure in [[Testing Policy]] and the
repository rules.
## See Also
- [Deployment Flow](../Architecture/Deployment%20Flow.md)
- [[index]]
- [Dankclaw deployment architecture](../Architecture/Network%20Architecture.md)
serverAppConfigs-docs/LLM/frank-archon.md
+41 -38
View File
@@ -1,7 +1,7 @@
---
type: reference
title: "frank-archon on Dankserver — radiance (StillDeadcode) 1.0.13, publisher Qwen3.8-Flash-Next iq4r .rad, 2× R9700 TP=2 (ACTIVE)"
description: "The one frank-archon page: the live radiance shape and harness contract (read off scripts/frank-archon.env), the Eras table carrying every rollback path including the re-download doors for the purged payloads, the laws this endpoint taught (pinned-RAM/TTM preflight, the KV dtype law, the one per-card VRAM budget, descriptor selection), the wire contract, and the files that own it. It replaces the nine per-era runbooks."
description: "The radiance endpoint: live shape and harness contract, every era with its rollback path, and the laws it taught."
tags:
- llm
- frank-archon
@@ -11,6 +11,7 @@ tags:
- r9700
- tp2
- expert-offload
- endpoint
timestamp: 2026-10-18
---
@@ -34,7 +35,7 @@ header carries the measured arithmetic beside each number).
`qwen4exp-w4nl64-i8-hc8m`): routed experts as 4-bit codes in a 16-level non-uniform codebook
(128-pt Walsh–Hadamard rotation, GPTQ on a 10M-token calibration + 3 coordinate-descent
sweeps, E4M3 scale per 64 weights); **10 whole bf16-protected experts per layer**; int8 W8A8
trunk + lm_head; fp8 HC mixtures; PLE n-gram fp8 at the checkpoint's fixed scale (47.7 GiB);
trunk + lm_head; fp8 HC mixtures; PLE n-gram fp8 at the checkpoint's fixed scale (47.7 GiB)
MTP head and the bf16 27-layer vision tower **in the container**. Card quality vs bf16: mean
KL 0.0789 all-tokens / 0.0239 assistant turns, top-1 92.85 %. The abliterated family lives on
[[frank-dragoon]]. Fetched by `scripts/download-frank-archon-iq4r.sh --fetch` (hfget resume +
@@ -52,7 +53,7 @@ command (the 2026-09-30 no-pipe/grep-race lesson) — so a retagged look-alike o
`latest` cannot satisfy it. 1.2.0 is a rejected serve pin (Laws); the
`--vram-weights-mib`/`--vram-kv-mib` pair is accepted by both pins, which is the rollback symmetry.
**Trio** — `scripts/frank-archon.env` (every number, with its arithmetic in the header),
**Trio** — `scripts/frank-archon.env` (every number, with its arithmetic in the header)
`scripts/serve-frank-archon.sh`, `scripts/frank-archon.service`. Gates run in order: image
digest present → `.rad` present and ≥ `RAD_MIN_MIB` (a truncated convert would otherwise buy a
guaranteed long-timeout boot instead of this fast failure) → template present → `/dev/kfd` +
@@ -63,10 +64,10 @@ the name registered, and every one of those identities holds this GPU. Container
(publisher-vetted, `docs/DOCKER.md`): `--network host`, `--init` (so the engine releases the
cards on SIGTERM), `--security-opt seccomp=unconfined` (the default profile refuses io_uring —
the PLE table's reads — and the NUMA calls the ROCm runtime makes for host memory), `--ulimit
memlock=-1:-1`, `/models` ro + `/data` rw + the tune cache at `/opt/radiance/share/radiance/tune`,
memlock=-1:-1`, `/models` ro + `/data` rw + the tune cache at `/opt/radiance/share/radiance/tune`
template as a single-file mount, then `exec podman wait`: the script's lifetime IS the
container's, so a crash is a nonzero `ExecStart` and the restart policy is real.
Unit: `After=network-online.target`, `WantedBy=multi-user.target`, `User=dank`,
Unit: `After=network-online.target`, `WantedBy=multi-user.target`, `User=dank`
`LimitMEMLOCK=infinity`, `StartLimitIntervalSec=0`, `OOMScoreAdjust=-500` (the last process the
OOM killer should pick), `Restart=always` **not** `on-failure` — a clean API-server exit after a
worker fault reads as success to systemd (the 2026-10-01 vLLM lesson) — and
@@ -83,7 +84,7 @@ on stop. Boot budget 1800 s (the publisher's compose allows a 15-minute `start_p
| VRAM pair | `VRAM_WEIGHTS_MIB=25600` + `VRAM_KV_MIB=5120` = 30,720 of the 30,969 ceiling; `GPU_HEADROOM_MIB=96` |
| Expert tier | `PLACEMENT=expert_tiered`, `HOST_POOL_MIB=44928` **per rank** |
| PLE | `NGRAM_PLACEMENT=ram` (47.7 GiB pinned; refuses boot on a short memlock) |
| Spec / batch | `SPEC_TOKENS=2` — the `.rad` carries the model's own `mtp.*` head (`mtp.fc_hidden`/`fc_embedding`/`draft_head`); the publisher's card ships depth 3 as its peak and this fleet's decode ladder was measured at 2. `MAX_BATCHED_TOKENS=2048` |
| Spec / batch | — |
| Prefix cache | `PREFIX_HOST_MIB=8192` + `PREFIX_DISK_MIB=65536` under `STATE_DIR=/home/dank/radiance-state` (mounted rw at `/data`) |
| Template | `TEMPLATE=scripts/qwen38-froggeric-v22.5.jinja` (28,234 B) via `--override-chat-template` — froggeric's fixed Qwen3.8 template (the repo root's `chat_template.jinja`, vendored); the checkpoint's own template is baked into the `.rad` by rad-convert and this overrides it, and radiance derives the reasoning and tool-call reply format from it |
| TTM gate | `TTM_MIN_MIB=150000`, `TTM_TARGET_MIB=204800` |
@@ -106,9 +107,9 @@ one string).
**Harness contract** (`/home/blank/.omp/agent/models.yml`) — provider `llamacpp (dankserver)`
→ `http://192.168.0.235:7770/v1`, `api: openai-completions`, `auth: none`, `id: frank-archon`
(= `FRANK_ARCHON_SERVED_NAME`; the provider *label* is legacy and renaming it breaks
`config.yml` references). `contextWindow: 200000` must equal `MAX_MODEL_LEN`;
`config.yml` references). `contextWindow: 200000` must equal `MAX_MODEL_LEN`
`maxTokens: 16384` (clamp `min(maxTokens, window − estimate − 4096)`; compaction fires at
85 % → 170,000 est × 1.03 + 16,384 ≤ 200,000); `input: [text, image]`;
85 % → 170,000 est × 1.03 + 16,384 ≤ 200,000); `input: [text, image]`
`compat.reasoningContentField: reasoning_content`; `reasoningEffortMap` minimal/low→low,
medium→medium, high/max→xhigh; `extraBody` temperature 1.0 / top_p 0.95 / top_k 20 / min_p 0 /
presence_penalty 0 / repetition_penalty 1.0 + `chat_template_kwargs {enable_thinking,
@@ -124,30 +125,34 @@ host-side unit drift is invisible to it.
## Eras
One row per retired era on this port. The **Rollback** cell is the load-bearing one: anything
marked "re-fetch" is gone from `/home/dank/models/`, which today holds only `StillDeadcode`,
marked "re-fetch" is gone from `/home/dank/models/`, which today holds only `StillDeadcode`
`rad` and `tpl.jinja`.
| Window | Payload | Runtime | Shape | Replaced because | Rollback |
|---|---|---|---|---|---|
| 2026-10-08 → 10-15 | in-house `qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB (Swift1.5 abliterated BF16, GPTQ w4 experts + 24 bf16 protected, KLD 0.0131 / top-1 97.92 %) | radiance 1.0.13, same digest | 3 × 200,000 fp8 KV, slab 25,600 / pool 5,120 — the shape the live file still carries | the engine author's own container is the supported path and his quality ceiling; the finetune's refusal-bending left the endpoint | `.rad` purged 2026-10-17 (trash-staged then emptied, Frank's approval). Re-download `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` → `qwen4exp-w4nl64-i8-hc8m.rad`, 122,071,973,888 B, LFS sha256 `09dd9306…`, via the iq4r fetch script's hfget/verify pattern retargeted at that repo; point `MODEL_FILE` at it, redeploy. Re-**convert** is dead: the BF16 source was `rm`'d 2026-10-15 (`kld-ref`, grams, corpora and `/home/dank/logs/swift15_*.log` survive) |
| 2026-10-05 → 10-08 | `qwen4exp-fp8-native.rad`, 177,109 MiB (173.0 GiB) — native modelopt block-FP8, no requantisation | radiance 1.0.7 line, same digest | TP=2 wht6, 200,000 × **2** slots (3 later), slab 26,071 + fp8 KV pool 3,543, host pool 40,960/rank, `--expert-vs-cache-ratio 0.82`, spec 3, MNBT 2048; that pair measured 30.08 t/s decode on the 2026-10-06 43.6k greedy probe and 4,379 t/s A-cold prefill on 2026-10-11 | native FP8 experts (56.4 GiB/rank) ate a 40 GiB pool and left KV at 3,543 MiB = 386,904 tokens, **1.93 sessions** — context economics, not quality; and the host pool could not be trimmed to pay for it (36,864 refused the boot) | rebuild-only: `scripts/download-frank-archon.sh` (official `Qwen/Qwen3.8-Flash-Next-FP8`, 131 shards, 172.8 GB, ungated) + `scripts/build-frank-archon-rad.sh` (no-recipe convert under the descriptor law); raw **and** `.rad` were `rm`'d 2026-10-15 (Frank's call, to free the volume for the iq4r download) |
| 2026-10-05 (same day) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` safetensors, 131 shards | FreeToken PR #587 head `77a7201` (`rocm714_torch214`), image `localhost/frank-primary-freetoken:pr587-77a7201` | ONE R9700; KV sized in tokens `299,968` (a multiple of 64 — the `qsa_sparse` backend pins page 64), 2 running requests, `--moe-cache-rate 0.10`; 17.1 t/s decode | TP=2 cannot boot on ROCm: the per-rank `pynccl.cu` JIT cannot resolve `cuda_runtime.h` for gfx1201, so the second card idled | git tag `frank-archon-freetoken-pr587-20261005` (trio + `scripts/docker/frank-primary-freetoken.Dockerfile`, which exists only there); image still in rootless storage; payload needs a re-fetch, and that repo 403s until the gate is accepted on the account |
| 2026-10-05 → 10-08 | `qwen4exp-fp8-native.rad`, 177,109 MiB (173.0 GiB) — native modelopt block-FP8, no requantisation | radiance 1.0.7 line, same digest | — | native FP8 experts (56.4 GiB/rank) ate a 40 GiB pool and left KV at 3,543 MiB = 386,904 tokens, **1.93 sessions** — context economics, not quality; and the host pool could not be trimmed to pay for it (36,864 refused the boot) | rebuild-only: `scripts/download-frank-archon.sh` (official `Qwen/Qwen3.8-Flash-Next-FP8`, 131 shards, 172.8 GB, ungated) + `scripts/build-frank-archon-rad.sh` (no-recipe convert under the descriptor law); raw **and** `.rad` were `rm`'d 2026-10-15 (Frank's call, to free the volume for the iq4r download) |
| 2026-10-05 (same day) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` safetensors, 131 shards | FreeToken PR #587 head `77a7201` (`rocm714_torch214`), image `localhost/frank-primary-freetoken:pr587-77a7201` | — | TP=2 cannot boot on ROCm: the per-rank `pynccl.cu` JIT cannot resolve `cuda_runtime.h` for gfx1201, so the second card idled | git tag `frank-archon-freetoken-pr587-20261005` (trio + `scripts/docker/frank-primary-freetoken.Dockerfile`, which exists only there); image still in rootless storage; payload needs a re-fetch, and that repo 403s until the gate is accepted on the account |
| 2026-09-29 → 10-05 (interrupted 09-30 → 10-01 by GLM) | same orcarouter FP8 safetensors | tcclaviger vLLM fork, `docker.io/tcclaviger/vllm@sha256:da402966…` | shape G: 3 × 200,704 fp8, KV 3,166,500,000 B/rank → 401,408 tokens, arena 17.77 GiB / 7,762 slots, util 0.982, MNBT 2048, no spec, no prefetch, prefix caching ON | radiance gave real TP=2 (no RCCL) and a 2-3 min boot, and the fork never fixed its long-context class | git tag `frank-archon-vllm-shapeG-20261003` (trio + the 362-line env lore); image still in dank's rootless storage; payload needs a re-fetch. The 2026-10-01 fp8+MTP4 rung is tag `frank-archon-fp8-mtp-20261001` |
| 2026-09-30 → 10-01 | `Intel/GLM-5.3-Flash-W4A16-AutoRound` ~170 GiB (sym int4 g128, `auto_gptq`) | same tcclaviger digest; `R4dW4a16MoEMethod` → libr4d grouped W4A16 | TP=2 + `--enable-expert-offload --expert-precise-memory`, 2 × 131,072 bf16 KV, util 0.98, MNBT 2048, 2,383 slots + 73.64 GiB pinned per rank; 9.9-10.0 t/s decode / 918 t/s prefill | the 35 t/s target is unreachable as the fork ships: 44 % of the ~93 ms/token budget is pointwise glue in an **uncompiled** GLM trunk — the expert gather is 1.05 % of device time | payload (170 GiB) + drafter (2.2 GiB) went to `~/.AGENTTRASH/20261001-glm-purge/` and were emptied; the cutover was reversed by `d91379c`, restoring the `2339fae` trio; `scripts/patches/glm53-*.patch` survive in-tree but none applies byte-exact since the 2026-10-10 rename rewrote their context lines |
| 2026-09-27 → 09-30 | unsloth `Qwen3.8-Flash-Next-GGUF` Q8_0, 175.30 GiB / 6 shards + blockfp8 MTP4 head + mmproj; PLE derived 54,400,261,120 B | R9V radiance-vLLM fork `drwolfen/radiance-vllm-qwen4exp`, docker bundle `2dac17a2`, behind the docker→podman shim | `MAX_MODEL_LEN=131072`, 80-slot tiered cache (the policy max), MTP4, PLE **pinned**, prefix caching ON from 09-28 | the tcclaviger fork took the port for the GLM W4A16 experiment; Q8_0 had halved expert residency and a 128K envelope | host state survives: `/home/dank/r9v-state/q8xl/setup.json`, the fork checkout `/home/dank/GitPackages/radiance-vllm-qwen4exp` (local-only commits `363d606`/`30d4a32`/`2de0357`) and `.podman-shim/docker`; payload re-fetch first |
| 2026-09-30 → 10-01 | `Intel/GLM-5.3-Flash-W4A16-AutoRound` ~170 GiB (sym int4 g128, `auto_gptq`) | same tcclaviger digest; `R4dW4a16MoEMethod` → libr4d grouped W4A16 | — | — | payload (170 GiB) + drafter (2.2 GiB) went to `~/.AGENTTRASH/20261001-glm-purge/` and were emptied; the cutover was reversed by `d91379c`, restoring the `2339fae` trio; `scripts/patches/glm53-*.patch` survive in-tree but none applies byte-exact since the 2026-10-10 rename rewrote their context lines |
| 2026-09-27 (same day) | `Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS`, 92.39 GiB + PLE 26.82 GiB (`34fa36f8…`) | same R9V bundle + shim | `MAX_MODEL_LEN=196608`, offload budgets 146.25 / 78.75 GiB, PLE residency `ssd` mmap, tool parser `qwen3_coder`, thinking on the `reasoning` (vLLM) channel | same-day flip back to the reference quant with the MTP4 head, at Frank's direction | `/home/dank/r9v-state/uncensored/` (`setup.json`, `restore-config-20260927.env`) is still on disk; the IQ4_XS payload was deleted in the 2026-10-17 cleanup → re-fetch, point `R9V_STATE_FILE` back, drop the two q8-only keys and the template keys |
| 2026-09-26 → 09-27 | unsloth Q8_0, 175.30 GiB / 6 parts + shared-Q8_0 MTP 2,786,568,256 B + `mmproj-BF16.gguf` 907,542,944 B | unsloth llama.cpp b11030-mix prebuilt (commit `6ba30d05b`), ROCm gfx120X | `-ts 1,1` layer split, routed experts **and** the PLE table computed in host RAM (`-ot exps=CPU,per_layer_token_embd.weight=CPU`), 262,144 × 1 slot, then the same-day reshape to 4 × 262,144 with V `q8_0` and hot expert blocks 5/3 | the R9V radiance-vLLM fork took the port at Frank's direction (true TP=2 plus MTP4) | trio at commit `4992e5a`+; the GGUF/MTP/mmproj set is no longer on disk → re-fetch, then redeploy |
| 2026-09-26 (same day) | `Lucebox/DeepSeek-V4.1-Flash-ROCMFP23-GGUF`, 377.2 GiB, sha256 `2daea6ca…` | `localhost/lucebox-rocm:gfx1201`, built from `Dockerfile.rocm` + 3 local patches | ONE R9700, `--max-ctx 131072`, `--chunk 8192`, q4_0 K+V, `--ds4-prefill sparse`, routed experts on CPU from the page cache | 377 GiB fits neither 63.7 GiB of VRAM nor 246 GiB of RAM → NVMe page-cache thrash; Qwen Flash-Next Q8 gave better quality, vision and 262k at comparable speed | image still in dank's rootless storage; payload gone; the quadlet and its entrypoint were deleted by `8500600` — `git show 8500600^:dankserver/frank-primary/quadlet/frank-primary.container` (and `…:dankserver/frank-primary/frank-primary-entrypoint.sh`), stop the root unit, start the user quadlet |
| ≤ 2026-09 — 4 earlier eras: payloads, shapes and every measured number destroyed 2026-10-19 as obsolete | — | — | — | — | —
Before 2026-09-26 the port belonged to the **`frank-medium`** service: [[frank-medium-radiance-mxfp4]]
(vllm-radiance TP=2 MXFP4, 2026-09-20 → 09-26) is that era's page and is **not** superseded here.
Before 2026-09-26 the port belonged to the **`frank-medium`** service, and its era
pages are **not** superseded here: is the locator for that
endpoint's whole shape history, including the unresolved end date of the
vllm-radiance era (this table opens on 09-26; the era page claims 10-10).
## Laws learned here
### The radiance engine
- **KV dtype law.** `--kv-cache-dtype` takes only `bf16|fp8` — ONE dtype for the whole cache,
so llama.cpp's exact-K / quantized-V pair (what [[frank-zealot]] runs as `--cache-type-k`/
a **radiance** restriction, not stock vLLM's: upstream advertises `auto|fp8|fp8_e5m2|fp8_e4m3`
(`docs.vllm.ai` engine-args) and even there the advertised set is not the usable set — several
accepted values crash engine init at the attention layer's own dtype check
([vllm-project/vllm#42587](https://github.com/vllm-project/vllm/issues/42587)).
So llama.cpp's exact-K / quantized-V pair (what [[frank-zealot]] runs as `--cache-type-k`/
`--cache-type-v`) is unservable here: the rung below bf16 is fp8 on both halves. Measured
per-rank cost out of the boot ledger's group table (fp8 boot 2026-10-08, bf16 boot
2026-10-11): `kv_attn` 13 layers / 4-token blocks pages **26.00 → 52.00 KiB**, while
@@ -188,7 +193,7 @@ Before 2026-09-26 the port belonged to the **`frank-medium`** service: [[frank-m
buffers + arena onto the weights side (~29.6 GiB/card effective).
- **Idle KV VRAM is the expert cache's spillover — the price is paid twice.** `mover: 2275
slots can be lent out of the KV cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <=
256 tokens`; the lend grows with the pool (2.69 GiB from 3.46, 6.08 from 6.912, 4.22 today),
256 tokens`; the lend grows with the pool (2.69 GiB from 3.46, 6.08 from 6.912, 4.22 today)
so a bigger KV pool steals weight residency from the slab AND from the lendable cache. Priors
on this card/model: stealing 2.14 GiB of weights to buy `MAX_MODEL_LEN 262144` cost 2.5 %
decode; the other direction (`--expert-vs-cache-ratio` 0.82 → 0.84, +2.5 GiB of weights) cost
@@ -268,7 +273,7 @@ Before 2026-09-26 the port belonged to the **`frank-medium`** service: [[frank-m
EMULATION backend an offload request of **118 GiB** landed as **~238 GiB** of real host RAM —
the dequantized copy rides on top of the pinned weights, and the doubling is invisible in the
weight count. Read the engine log for `Using 'EMULATION' Mxfp4 MoE backend` before concluding
"doesn't fit"; the container-side procedure and the kernel gate are [[RDNA4 Container Serving]].
"doesn't fit"; the container-side procedure and the kernel gate are .
- **One encoding per expert table** (`rad_arch_declare`): a whole expert kept bf16 is the only
thing legal beside the table, so a split tier (down=fp8, gate_up=w4) is unservable.
- **Descriptor-selection law.** The plugin is picked by the exact `(arch_id, quant descriptor)`
@@ -284,7 +289,7 @@ Before 2026-09-26 the port belonged to the **`frank-medium`** service: [[frank-m
write temp + `os.replace`.
- **Convert is card-less** (`rad-convert` declares on CPU, GPTQ solves on CPU BLAS) and
`--plan-only` goes first: a plan that resolves every op will boot. `rad-tune` must be
declared at the shapes you serve and its cache lives in the *image* (`$RADIANCE_HOME/tune`),
declared at the shapes you serve and its cache lives in the *image* (`$RADIANCE_HOME/tune`)
so it only survives on the `$FRANK_ARCHON_STATE_DIR/tune` bind-mount the serve script adds.
- **Quant gates are scored at the production shape** (MNBT 2048, not the recording's 32768)
against the BF16 container: KL ≤ 0.03, top-1 ≥ 95 %, and the verdict is chunk-size invariant
@@ -349,7 +354,7 @@ Before 2026-09-26 the port belonged to the **`frank-medium`** service: [[frank-m
- **TP=2 here** is legal since the 2026-09-20 ProArt B850 swap (x8/x8 on CPU root ports,
`Routing+`/`ReqEn+` end to end) only because radiance's wire is `libr4d`'s own one-shot
all-reduce; RCCL needs atomics on the whole chain and fails `hipErrorIllegalState` without
them ([[Dual-R9700-Cooperation]]). `wht6` buys +5.6-6.3 % prefill for ~1.3 % of 8-stream
them (). `wht6` buys +5.6-6.3 % prefill for ~1.3 % of 8-stream
decode, `exact` is the bit-exact alternative.
- `--prefix-cache-dir` creation is **not recursive** — the serve script `mkdir -p`s the path or
the disk tier silently never opens. The tier does survive a payload swap when arch and
@@ -357,7 +362,7 @@ Before 2026-09-26 the port belonged to the **`frank-medium`** service: [[frank-m
### Dead ends — do not re-try
- **radiance 1.2.0 as the serve pin** (`sha256:665775ed1d90de6dbb25558e07fc60d20328f1346f96f3c6e00845eb763f01fa`,
- **radiance 1.2.0 as the serve pin** (`sha256:665775ed1d90de6dbb25558e07fc60d20328f1346f96f3c6e00845eb763f01fa`
commit `e95d397`), benched 2026-10-06 on this exact shape with a 43.6k greedy probe: decode
29.21 t/s and draft acceptance 0.7123 vs the 1.0.13 control's 30.08 / 0.7549, and the control
reproduced the recorded 0.755 exactly — greedy acceptance is deterministic, so the drop is a
@@ -368,7 +373,7 @@ Before 2026-09-26 the port belonged to the **`frank-medium`** service: [[frank-m
both pins accept. Re-test 1.2.0+ once the rotation change is untangled.
- **FreeToken TP=2 on ROCm:** the `pynccl.cu` JIT cannot resolve `cuda_runtime.h` for gfx1201 —
an untested edge of that PR, not a board fault.
- **tcclaviger forks:** `--max-num-seqs 2` (`ConstraintViolationError` on `query_start_loc`);
- **tcclaviger forks:** `--max-num-seqs 2` (`ConstraintViolationError` on `query_start_loc`)
`--gpu-memory-utilization 1.00` unbootable, 0.982 is the law; expert prefetch is poison (36 %
precision — deleting it was +40-51 % decode); TunableOp-with-offload corrupts numerics
(repetition loops, early EOS) — autotuning that changes text is broken tuning; ngram-only
@@ -388,12 +393,12 @@ Before 2026-09-26 the port belonged to the **`frank-medium`** service: [[frank-m
`-ot` granularity the lever is COUNT.
- **Never mount a diagnostic hook on `PYTHONPATH` without a marker guard:** `sitecustomize` is
imported by every interpreter, a child inherits the hook and spawns again — a 19-level python
fork bomb inside the container. Kill with `podman rm -f` (the cgroup owns every generation),
fork bomb inside the container. Kill with `podman rm -f` (the cgroup owns every generation)
never pattern-kill an interpreter name on this host; use `TORCH_LOGS` for dynamo questions.
## Contract (the wire)
- **Live surface:** `/health` (== `/ping`), `/v1/models`, `/v1/chat/completions`,
- **Live surface:** `/health` (== `/ping`), `/v1/models`, `/v1/chat/completions`
`/v1/completions`, `/metrics`, `/stats`, `/sessions`, `/dashboard`. `/health` answers 200
JSON (model id, `in_flight`, `queue_depth`) while serving and **503 only while stopping**
(`core/server/admin.cpp:385`), and the listener binds *after* the full load (`main.cpp`:
@@ -430,12 +435,12 @@ Before 2026-09-26 the port belonged to the **`frank-medium`** service: [[frank-m
- **Observability:** `/metrics` carries vLLM-compatible names plus `radiance:*`, scraped by the
Dankclaw Prometheus job at `192.168.0.235:7770` over plain LAN routing (the netavark
published-port quirk is same-host only). The boot ledger — the referee for every shape number
above — is in the container log (`podman logs frank-archon`);
above — is in the container log (`podman logs frank-archon`)
`/home/dank/logs/frank_archon.log` is where the serve script redirects `podman run`.
## Files
- Shape and law owner: `scripts/frank-archon.env` (measured arithmetic in its header),
- Shape and law owner: `scripts/frank-archon.env` (measured arithmetic in its header)
`scripts/serve-frank-archon.sh` (gates image / `.rad` size / template / GPU nodes, retires
stale containers, `podman wait`), `scripts/frank-archon.service`, `scripts/ttm-preflight.sh`.
- Payload doors: `scripts/download-frank-archon-iq4r.sh` (live), `scripts/download-frank-archon.sh`
@@ -443,21 +448,19 @@ Before 2026-09-26 the port belonged to the **`frank-medium`** service: [[frank-m
rebuild), `scripts/download-frank-archon-swift15.sh` (the BF16 source a w4dn8 re-convert
needs), `scripts/hfget.sh` / `scripts/hf-fetch.sh`; template
`scripts/qwen38-froggeric-v22.5.jinja`.
- Convert campaign (dormant, kept for a future quant): `scripts/convert-swift15-w4dn8.sh`,
`scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`, `scripts/run-swift15-bf16.sh`,
`scripts/kld-gate-swift15.sh`, `scripts/kld-then-calib.sh`, `scripts/calib-push.py`,
`scripts/make-calib-corpus.py`, `scripts/derive-protected-experts.py`,
- Convert campaign (dormant, kept for a future quant): `scripts/convert-swift15-w4dn8.sh`
`scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`, `scripts/run-swift15-bf16.sh`
`scripts/kld-gate-swift15.sh`, `scripts/kld-then-calib.sh`, `scripts/calib-push.py`
`scripts/make-calib-corpus.py`, `scripts/derive-protected-experts.py`
`scripts/apply-protected-cast.sh`, `scripts/verify-swift15-ple.py`.
- Probes: `scripts/probe-frank-archon-kv.py` (reads slots/window/dtype from the deployed env so
it cannot drift; leg D = decode), `scripts/frank-archon-expert-activity.py`,
it cannot drift; leg D = decode), `scripts/frank-archon-expert-activity.py`
`scripts/autoresearch-frank-archon.sh` + `scripts/autoresearch/frank-archon-measure.py`.
- Deploy and tests: `scripts/deploy.sh` / `scripts/deploy-remote.sh` (`deploy_llm_archon`),
- Deploy and tests: `scripts/deploy.sh` / `scripts/deploy-remote.sh` (`deploy_llm_archon`)
`tests/frank_archon_unit.bats` (the seams gating the KV-pool arithmetic, the VRAM pair and the
ttm gate), `tests/changed_tests.bats`, `tests/deployment_scoping.bats`.
- Retired-era leftovers in-tree: `scripts/patches/glm53-*.patch`,
- Retired-era leftovers in-tree: `scripts/patches/glm53-*.patch`
`scripts/patches/gpu_worker-kv-cap.patch` (fork archaeology; none applies byte-exact today).
Related: [[frank-dragoon]] (the abliterated family), [[frank-zealot]],
[[Dual-R9700-Cooperation]], [[deployment]], [[swift15-campaign-worklog]],
[[dankserver-frontier-moe-feasibility]], [[x4-pair-experiment]], [[RDNA4 Container Serving]],
[[frank-medium-radiance-mxfp4]], [[LLM/index]].
---
serverAppConfigs-docs/LLM/frank-dragoon.md
+21 -30
View File
@@ -1,7 +1,7 @@
---
type: reference
title: "frank-dragoon on Dankclaw — halogen-flash-server 0.16.2, native v2 abliterated Qwen3.8-Flash-Next (canonical)"
description: "The one runbook for frank-dragoon, the Strix Halo iGPU endpoint (192.168.0.143:7770): current shape from the env, every retired era with its rollback path, the durable laws this endpoint paid for, the wire contract, and the files that own it."
description: "The iGPU endpoint: live shape, every era with its rollback path, the laws it paid for, and the files that own it."
tags:
- llm
- frank-dragoon
@@ -9,6 +9,7 @@ tags:
- strix-halo
- gfx1151
- halogen
- endpoint
timestamp: 2026-10-18
---
@@ -53,7 +54,7 @@ unauthenticated token port 8730 stays on container loopback. Engine stdout →
~20 GiB above the 20 GiB reserve, against 10–15 GiB of baseline co-tenancy (frank-zealot, ComfyUI,
Caddy/observability).
- Harness (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw)`): `contextWindow: 155648`
(= ceiling − 45k per the 400-zone law, never higher), `maxTokens: 45056`, `input: [text, image]`,
(= ceiling − 45k per the 400-zone law, never higher), `maxTokens: 45056`, `input: [text, image]`
`reasoningContentField: "reasoning_content"`, `high→xhigh` fold, `extraBody` 1.0/0.95. Compaction
rides [[frank-zealot]].
- Deploy `./scripts/deploy.sh dankclaw --service frank-dragoon`; gate = boot-ledger greps (weights,
@@ -69,28 +70,19 @@ rollback plan — recoverable only until the trash is purged.
| Window | Payload | Runtime | Shape | Replaced because | Rollback |
|---|---|---|---|---|---|
| 2026-10-06 → 10-17 | Swift1.5 abliterated v2 `.hgn` r2 66,687,678,464 B `9c880b8b…` + same trio | flash-server 0.16.2, same digest | 2 slots, ctx 262,144, pool 524,288 (14.6 GiB), MTP always-on | 2026-10-17 A/B at that shape: NIAH 18/18 vs 17/18, refusal census 0/5 on both, tool calls + red-PNG probe identical, needle hit at both depths of a 21.5k haystack, decode 50.0 vs 46.8 t/s, cold prefill 1,604 vs 1,425 t/s, PPL 3.968 vs 3.877 (finetune's edge gone: +2.3 %), KL vs the swift15 logits mean 0.155 / median 0.057 with top-1 agreement 85.1 % — two sound models disagreeing on content, not a corrupt forward pass, thinking 1,476 vs 784 mean, 7.4 GiB RAM back | `/home/dank/models/halogen-flashnext-v2/` + `scripts/download-frank-dragoon-flashnext.sh` |
| 2026-10-06 → 10-17 | Swift1.5 abliterated v2 `.hgn` r2 66,687,678,464 B `9c880b8b…` + same trio | flash-server 0.16.2, same digest | 2 slots, ctx 262,144, pool 524,288 (14.6 GiB), MTP always-on | — | `/home/dank/models/halogen-flashnext-v2/` + `scripts/download-frank-dragoon-flashnext.sh` |
| 2026-10-05 → 10-06 | peonist `halogen-qwen3.8-27b` `qwen3.8-27b-p1w4d-d2.hgn` 35,865,565,184 B `274c3fc7…` | halogen 0.1.4 `sha256:dc0a39a0…` | 1 slot × 200,700, DFlash2 drafter, `HALOGEN_CACHE_MB=16384`, text-only, censored base | flash-server v2 restored vision, abliteration and MTP | `/home/dank/models/halogen-qwen3.8-27b/` + `scripts/download-frank-dragoon-halogen.sh`; 0.1.4 digest still in podman storage |
| 2026-10-03 → 10-05 | kingjones777 Uncensored `Q6_0_ROCMFPX_AGENT.gguf` 24,856,607,008 B `e966deb3…` + `mtp-Q4_0` `051a1764…` + BF16 mmproj | `kingjones30/ROCmFPX` @ `36e9acd` (`scripts/build-rocmfpx.sh`) | 1 × 262,144 f16 KV, `draft-mtp` n-max 4, vision on | halogen card claimed 31.7 t/s vs 27.6–28.1 measured | `/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/` + `/home/dank/GitPackages/kingjones-ROCmFPX/build` + `download-frank-dragoon-kingjones-uncensored.sh` |
| 2026-10-03 → 10-05 | kingjones777 Uncensored `Q6_0_ROCMFPX_AGENT.gguf` 24,856,607,008 B `e966deb3…` + `mtp-Q4_0` `051a1764…` + BF16 mmproj | `kingjones30/ROCmFPX` @ `36e9acd` (`scripts/build-rocmfpx.sh`) | 1 × 262,144 f16 KV, `draft-mtp` n-max 4, vision on | — | `/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/` + `/home/dank/GitPackages/kingjones-ROCmFPX/build` + `download-frank-dragoon-kingjones-uncensored.sh` |
| 2026-10-02 → 10-03 | aligned `Q8_0_ROCMFPX_AGENT.gguf` 28,214,425,952 B `f3ef2bdb…` | same fork pin | 1 × 262,144 f16 KV, `draft-mtp` 4 (MTP +123 % decode) | the uncensored file was faster *and* uncensored | `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/` + `download-frank-dragoon-kingjones.sh` |
| 2026-10-01 (one day) | MiMo-V2.6-Distill-Qwen-9B BF16 17.53 GiB + cross-model MTP from `Qwen3.5-9B` | vLLM 0.30 TheRock `kyuz0/vllm-therock-gfx1151` `sha256:98c1f3ac…` | 4 seqs × 262,144 on a scheduler-shared pool, 2 draft tokens | 6.19 t/s with MTP / 9.01 without vs gufo ~15, and no ≥6-bit vLLM quant path boots on gfx1151 | staged `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` until purged; fetch trio in `old/` |
| 2026-09-28 → 10-01, again 10-01 → 10-02 | Swift-1.5 Uncensored `UD-Q8_K_XL` 31,457,990,272 B `9076f99a…`, then `UD-Q4_K_XL`; z-lab DFlash2 Q4_K_M | gufo v0.1.1 (`b0f8673`), source build | 1 × 262,144, `--draft-policy adaptive --draft-tokens 7`, no KV knob, no Jinja | ROCmFPX gave 8.39 bpw + MTP at 17 t/s | payloads **deleted 2026-10-03** → re-download `download-frank-dragoon-swift.sh` (+ `download-frank-dragoon.sh` for the drafter); tag `frank-dragoon-gufo-q4kxl-20261001`; `scripts/build-gufo.sh` |
| 2026-10-01 (one day) | MiMo-V2.6-Distill-Qwen-9B BF16 17.53 GiB + cross-model MTP from `Qwen3.5-9B` | vLLM 0.30 TheRock `kyuz0/vllm-therock-gfx1151` `sha256:98c1f3ac…` | 4 seqs × 262,144 on a scheduler-shared pool, 2 draft tokens | — | staged `/home/dank/.AGENTTRASH/20261002-retired-llm-payloads/` until purged; fetch trio in `old/` |
| 2026-09-28 → 10-01, again 10-01 → 10-02 | Swift-1.5 Uncensored `UD-Q8_K_XL` 31,457,990,272 B `9076f99a…`, then `UD-Q4_K_XL`; z-lab DFlash2 Q4_K_M | gufo v0.1.1 (`b0f8673`), source build | 1 × 262,144, `--draft-policy adaptive --draft-tokens 7`, no KV knob, no Jinja | — | payloads **deleted 2026-10-03** → re-download `download-frank-dragoon-swift.sh` (+ `download-frank-dragoon.sh` for the drafter); tag `frank-dragoon-gufo-q4kxl-20261001`; `scripts/build-gufo.sh` |
| 2026-09-21 → 09-28 | unsloth `Qwen3.8-27B-GGUF:UD-Q8_K_XL` 31.46 GiB + DFlash2 Q4_K_M 1.14 GiB | unsloth llama.cpp `b11030` prebuilt | 1 × 262,144, q8_0 KV, Qwen-Sharp v22.4.0 template | gufo: +37 % prefill shallow, +80 % at 131k, 4.5× faster load | `/home/dank/GitPackages/unsloth-llama-b11030/` on disk; payload deleted 2026-10-03 → `download-frank-dragoon-unsloth.sh` |
| 2026-09-21 (same day) | unsloth `Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS` 87.3 GiB in 3 split parts | same prebuilt | 1 × 200,000, MTP depth 2 via explicit `-md` shared Q8_0 head | swapped to the 27B dense hours later | payload + prebuilt on disk at retirement; the parts ARE the payload (concatenating → `invalid split file name`) |
| 2026-09-21 (same day) | `jcbtc/Qwen3.8-27B-IU4-Kairic-Edge` + 3 `.pfs` = 26.05 GiB | ciru-ai/ROCmFPX `kairic-edge-qwen38-27b-v1.2` @ `205a3e5` | 1 × 262,144, native MTP 4 + n-gram | same-day swap to the 27B dense | payload + runtime intact on disk at retirement |
| 2026-09-19 → 09-21 | `jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca` | CIRU v4.4.1 (QSA window hotfix) | never written down — that runbook is a frontmatter stub | the stock llama.cpp prebuilt won | `scripts/download-frank-dragoon-orca.sh` |
| 2026-09-11 → 09-15, again 09-16 → 09-19 | `peonist-ai/halogen-qwen3.8-flash-next` w4b 115.55 GiB + in-place-versioned `overlay.hgn` | halogen-flash-server 0.5.8, rootless Quadlet | 2 × 200,192 over a 400,384 pool, `MAX_TOK 24576`, `HOST_RESERVE 20`, vision OFF | displaced by CIRU Orca; the shape starved co-tenants | re-tag 0.5.8 + first-start `HALOGEN_DOWNLOAD` re-fetch into `/home/dank/models/halogen`; the Quadlet left the repo 2026-10-05, so the unit must be rewritten |
| 2026-09-15 → 09-16 (one day) | `mradermacher/Qwen3.8-Flash-Next-Uncensored` i1-Q4_K_M 111 GiB + f16 mmproj | ik_llama.cpp `d85a1d7`, 3090 + CPU hybrid | 1 × 262,144, `-ot exps=CPU`, `GGML_CUDA_NO_PINNED_WEIGHTS=1` | 3.8–4.0 t/s: ~100 CPU↔GPU graph splits per token, GPU at 5 % SM | `/home/dank/models/mradermacher-Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`, `/home/dank/GitPackages/ik_llama.cpp/build`, trio `scripts/.AGENTTRASH/frank-dragoon.{env,service}.ik-llama.cpp` + `serve-frank-dragoon.sh.ik-llama.cpp` |
| 2026-08-30 → 09-11 | kingjones STRIX `Q4_0_ROCMFP4-STRIX.gguf` 113.47 GiB (51.2 B PLE via mmap) | ROCmFPX @ `36e9acd` | 2 × 200,704 over one 401,408 f16 window, spec off, f16 mmproj | halogen-flash-server ≈ 4× prefill | payload dir + `/home/dank/GitPackages/kingjones-ROCmFPX/build`; trio `.AGENTTRASH/frank-dragoon.{env,service}.llama.cpp` + `serve-frank-dragoon.sh.llama.cpp` |
| 2026-08-25 → 08-29 (restored 08-29 → 08-30 after the Q8_0 AGENT file crashed before it ever served) | `kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP`: `Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf` 23.15 GiB (private ftype 114) + a **SEPARATE** MTP head `mtp-Qwen3.8-27B-Q4_0.gguf` 1.56 GiB via `--model-draft` + `mmproj-Qwen3.8-27B-Q8_0.gguf` 0.59 GiB | CIRU ROCmFPX llama.cpp (`build-promptforge`) | 1 slot at `-c 196608`, `-b 8192 -ub 8192 -ctxcp 32`, `-fa on --no-mmap`, f16 K+V, `draft-mtp` n-max 4 with the head on ROCm1 at f16 KV, QwenSharp template (`enable_thinking`, `preserve_thinking`) behind the `peg-native` + `deepseek` chat/reasoning pairing the Pi agent expected | the Q8_0 AGENT experiment, then the STRIX Q4_0 tier | `/home/dank/models/kingjones-Qwen3.8-27B-STRIX-MTP/` (six card files + projector), runtime `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` |
| 2026-08-29 → 08-30 (one day) | agentionai FAST-imatrix **`v2/` joined-PLE** file 87.06 GiB | charlie12345/ROCmFPX `build-qwen38` + `scripts/patches/qwen4exp-qsa-f32-mask.patch` | 2 × 200,704 / 401,408 f16 KV, no MTP head loads | kingjones STRIX on its own pinned fork | `/home/dank/GitPackages/ROCmFPX/build-qwen38` + the agentionai payload, both on disk |
| 2026-08-18 → 08-25 | caf-eemin `Q4_0_ROCMI4.gguf` 13.50 GiB `ad51ecbe…` with embedded NextN/MTP | charlie12345/ROCmFPX, ROCmI4 W4A4 | one window, MTP `n4 / p0.60` | the 27B STRIX tier took the slot | re-fetch into `/home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP/` |
| through 2026-08-24 | CIRU ActiveFPX PromptForge v3: GGUF 15.44 GiB + 3 `.pfs` + mmproj ≈ 36.7 GiB | CIRU ROCmFPX `build-promptforge` | 1 × 262,144, native MTP depth 4, ngram off | eemin ROCmI4 | `/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/` + `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`; scripts `git revert 4c43839 7aee1df 40d50fe` or checkout `132e147` |
| 2026-08-22 (same day) | `jcbtc/Qwen3.8-27B-IU4-Kairic-Edge` 15.48 GiB + 3 `.pfs` | ciru-ai/ROCmFPX tag `kairic-edge-qwen38-27b-v1` @ `e97b324` | one window, IU4 W4A4 lane, ubatch 512 | prefill 1.5–2.3× faster but prose decode 15–26 % slower → back to v3 the same day | `/home/dank/models/Kairic-Qwen3.8-27B-IU4-Edge/` + `/home/dank/ciru-qwen38/ROCmFPX-kairic/build-kairic/bin/llama-server` |
| 2026-08-21 (same day) | unsloth `UD-Q8_K_XL` 31.4 GiB + z-lab DFlash2 Q4_K_M 2.0 GiB | upstream llama.cpp PR #27342, `GGML_VULKAN=ON` | Vulkan0 — HIP has no `TOP_K`, so DFlash could not run there | slower than the CIRU runtime it replaced | both files in `/home/dank/models/`; the PR build must be rebuilt |
| ≤ 2026-09 — 11 earlier eras: payloads, shapes and every measured number destroyed 2026-10-19 as obsolete | — | — | — | — | —
Port note: this slot settled on **:7770** on 2026-08-22 (52625 before it); the 2026-08-17 fleet redo
created it — earlier topologies in [[frank-redo-2026-08-17]].
created it — earlier topologies in .
## Laws learned here
@@ -133,7 +125,7 @@ created it — earlier topologies in [[frank-redo-2026-08-17]].
85–180 s until `HALOGEN_CACHE_MB=16384` was pinned. What the pool takes is what the table loses, so
re-measure the warm ratio after any pool change.
- **400-zone law:** the harness window MUST sit ~45k under the engine ceiling (`contextWindow:
155648` = 200,704 − 45,056) because the harness clamps `max_tokens` to `contextWindow − est − 4096`,
155648` = 200,704 − 45,056) because the harness clamps `max_tokens` to `contextWindow − est − 4096`
its chars/4 `est` under-counts code/agent traffic by ~3 %, and compaction only fires at 85 %. The
3 % is measured, not theory: 2026-10-07 at the old 200,192 ceiling, est 152,938 vs server 157,388 —
over by 354, a 400. At the live shape compaction fires at 132,301 est ≈ 136k measured (the 180k
@@ -179,9 +171,9 @@ created it — earlier topologies in [[frank-redo-2026-08-17]].
Gated-DeltaNet layers it also disables MTP's rollback path, −7…−15 %); DFlash across parallel slots
(shared draft ctx spuriously emits EOG); `-ot`/`-ncmoe` on a hybrid split (disables mmap, page-locks
~108 GiB and stranded the host until a power cycle); `--no-mmap` on any PLE-table file (silent cgroup
OOM-kill); forcing PLE to CPU (23 → 13.4 t/s); `HALOGEN_FA_TQ=256` (no decode win, −9.5 % prefill);
OOM-kill); forcing PLE to CPU (23 → 13.4 t/s); `HALOGEN_FA_TQ=256` (no decode win, −9.5 % prefill)
big ubatches as a decode knob (prefill-only, ub 512 fastest); Flash-Next MTP heads on the qwen4exp
loader (only `blk.48`, never the full per-layer norm set); mradermacher i1 GGUFs (NextN stripped);
loader (only `blk.48`, never the full per-layer norm set); mradermacher i1 GGUFs (NextN stripped)
vLLM quant paths on gfx1151; a custom Jinja template on gufo (no Jinja engine — send the `terse` text
as a client system message instead).
- Build lore if a fork is rebuilt: purge Ubuntu `librocwmma-dev` (shadows AMD's headers, `-I` cannot beat
@@ -190,7 +182,7 @@ created it — earlier topologies in [[frank-redo-2026-08-17]].
## Contract (wire)
- Live: `/v1/models`, `/health`, `/metrics`, POST `/v1/chat/completions`; `404`: `/ping`, `/props`,
- Live: `/v1/models`, `/health`, `/metrics`, POST `/v1/chat/completions`; `404`: `/ping`, `/props`
`/completion`, `/tokenize`. **No `Server-Timing` header and no `/completion` timing object** — measure
wall-clock plus `usage.prompt_tokens`.
- Thinking lands in `reasoning_content` and **counts against `max_tokens`**: a budget that dies
@@ -199,7 +191,7 @@ created it — earlier topologies in [[frank-redo-2026-08-17]].
thinking tokens mean on the 2026-10-17 battery, against 784 for the retired Swift1.5 finetune (×1.9)
and 905 for the censored 27B.
- `reasoning_effort` accepts only `xhigh|medium|low` (anything else is a template error — the
`models.yml` fold stays) and there is **no `"none"`**: disable thinking with `enable_thinking:false`,
`models.yml` fold stays) and there is **no `"none"`**: disable thinking with `enable_thinking:false`
which is also what the llama.cpp-era QwenSharp template wanted (`reasoning_effort:"none"` threw
there too). `preserve_reasoning` is a 400 on this engine; the llama.cpp-era key was
`chat_template_kwargs.preserve_thinking`, which decided whether `<think>` blocks stayed in recorded
@@ -213,7 +205,7 @@ created it — earlier topologies in [[frank-redo-2026-08-17]].
model answers "video", ~2,040 tokens per image, `http(s)` image URLs refused on purpose.
- The first request after a boot is a warmup artifact and may sit silent 45–60 s with 503
`engine_unresponsive` then self-recover — do not restart over it; `/health` 503 during load is loading.
Flags read once at startup, all bitwise except `HALOGEN_CK_OVERLAY`, `HALOGEN_MATMUL_TUNING_FILE`,
Flags read once at startup, all bitwise except `HALOGEN_CK_OVERLAY`, `HALOGEN_MATMUL_TUNING_FILE`
`HALOGEN_INDEXER_BUDGET`, `HALOGEN_COMPOSABLE_CONTEXT`.
- Deploy waits for :7770 to be free: a rootless container binds it at CREATE and exits 126, and a
leftover Quadlet beside the root unit guarantees the race.
@@ -224,13 +216,12 @@ created it — earlier topologies in [[frank-redo-2026-08-17]].
`podman run`, ledger to `/home/dank/logs/frank_dragoon.log`) · `scripts/frank-dragoon.service`
(`User=dank`, `LimitMEMLOCK=infinity`, `Restart=always`, `OOMScoreAdjust=-500`, `ExecStop=podman stop
-t 60`) → `deploy_llm_dragoon` in `scripts/deploy-remote.sh`.
- Fetchers: `download-frank-dragoon-v2abl.sh` (live); era rollbacks `-flashnext`, `-halogen`,
`-kingjones`, `-kingjones-uncensored`, `-swift`, `-unsloth`, `-q4`, `-orca`,
- Fetchers: `download-frank-dragoon-v2abl.sh` (live); era rollbacks `-flashnext`, `-halogen`
`-kingjones`, `-kingjones-uncensored`, `-swift`, `-unsloth`, `-q4`, `-orca`
`download-frank-dragoon.sh`; runtime pins `build-rocmfpx.sh`, `build-gufo.sh`.
- Verification: `tests/frank_dragoon_unit.bats` (payload gates, deployer/serve parity, spec block, KV
band), `tests/llm_units.bats`, `tests/llm_secrets.bats`, `scripts/census-frank-dragoon.sh`,
`scripts/battery-frank-dragoon.py`, `scripts/stage-frank-dragoon-eval.sh`,
band), `tests/llm_units.bats`, `tests/llm_secrets.bats`, `scripts/census-frank-dragoon.sh`
`scripts/battery-frank-dragoon.py`, `scripts/stage-frank-dragoon-eval.sh`
`scripts/trial-frank-dragoon.sh`.
- Related: [[deployment]] · [[Model Swap and Rollback]] · [[frank-zealot]] · [[frank-archon]] ·
[[dankclaw-iommu-translation-off]] · [[frank-redo-2026-08-17]] · [[Dankclaw System Specs]] ·
[[DFlash DSpark Speculative Decoding]] · [[qwen3.8-spec-matrix]] · [[qwen3.8-strix-optimizations]].
-Related: · · · ·
---
serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md
-225
View File
@@ -1,225 +0,0 @@
---
timestamp: 2026-09-17
type: reference
title: frank-medium — Batch and Ubatch Sweep
description: llama-bench sweep of --batch-size against --ubatch-size on frank-medium (2x Radeon AI PRO R9700, layer split) at the deployed context shape, and the rebalance it forced. Prefill peaks at ubatch 512 and falls on BOTH sides.
tags:
- benchmark
- llamacpp
- ubatch
- frank-medium
- dankserver
- layer-split
---
# frank-medium — Batch and Ubatch Sweep
Run date: 2026-09-16. Harness: `scripts/bench-batch-ubatch.sh` (one
`llama-bench` invocation per cell, a VRAM sampler per cell, the service stopped
for the duration and restarted on exit). Tested at the deployed shape: 262144
ctx, two slots, `--split-mode layer`, f16 KV, flash attention on, DFlash2
loaded by the service but absent from the bench.
The question was how large the logical batch and the micro-batch may be, and
what each buys. The answer is that `--ubatch-size` has an **optimum** on this
stack, that the logical batch does not matter at all, and that the optimum is
much cheaper than what was deployed — which then moved the layer split.
The deployment this sweep tuned (`frank-medium`, 2x R9700, llama.cpp layer
split) is retired; its shape and boot ledger are
[[frank-medium-llamacpp-dflash2]]'s, and the endpoint lineage continues on
[[frank-archon]]. What this page owns is the sweep, the harness traps and the
coupling law between batch and split.
## The shape of the measurement
A cell is a **64k prefill measured with 64k of KV already resident**, plus a
128-token decode at that same depth:
```
llama-bench -m <UD-Q8_K_XL> -p 65536 -n 128 -d 65536 -b <b> -ub <ub> \
-ngl 999 -dev ROCm0/ROCm1 -sm layer -ts 36/29 \
-fa on -ctk f16 -ctv f16 -r 3 --no-warmup
```
`-d 65536` is what puts depth in front of the measurement: a `tg` instance runs
with `n_prompt = 0`, and an unfilled `-d` would measure a cold cache. The depth
fill itself is excluded from the timing (and cached across reps), so the number
is "prefill a 64k prompt into a 64k conversation", which is the agent-loop shape
rather than a cold start.
Three `llama-bench` properties shaped the harness, all of them traps:
- **No `-c`**: the context is `n_prompt + n_gen + n_depth`, so a bench can never
decide whether a configuration's buffers fit beside the *deployed* KV.
- **A test that cannot allocate calls `exit(1)`**, taking the process with it —
hence one invocation per cell.
- **`/` separates devices within a configuration and `,` separates sweep
values**, so `-ts 36,29` would have benched each card on its own.
## Results — prefill, 64k depth (tok/s)
| `--batch-size` | ub 256 | ub 384 | **ub 512** | ub 768 | ub 1024 | ub 1280 | ub 1536 | ub 1792 | ub 2048 |
|---|---|---|---|---|---|---|---|---|---|
| 2048 | — | — | **1193.9** | 1147.9 | 1151.9 | 1141.7 | 1170.2 | 1128.8 | 1077.5 |
| 4096 | — | — | **1193.3** | 1129.6 | 1151.8 | 1143.7 | 1161.9 | 1151.9 | 1077.4 |
| 6144 | 1046.0 | 1068.0 | **1192.9** | 1135.0 | 1151.9 | 1151.4 | 1165.0 | 1147.7 | 1077.6 |
| 8192 | — | — | **1193.2** | — | — | — | — | — | — |
Per-cell standard deviations are 0.5–2.3 tok/s, i.e. under 0.2 %. **Decode is
15.87–15.89 tok/s on every single cell**, and the VRAM columns depend only on
the ubatch (2048 / 4096 / 6144 at a fixed ubatch report identical free space).
## Results — the same shape with no depth in front of it
`N_DEPTH=0`, same 64k prompt, two points for contrast:
| `--batch-size` | ubatch | prefill (tok/s) | decode (tok/s) |
|---|---|---|---|
| 6144 | 256 | 1543.5 ± 13.4 | 17.80 |
| 6144 | 1536 | 1722.7 ± 6.2 | 17.80 |
The **ubatch curve has the same shape at both depths** (below 512 is slower),
and the two tables differ by a uniform ~32 %: that is what attending over 64k of
resident KV costs. Depth is a tax on every configuration, not something the
batch knobs can trade against.
## What the data says
- **`--ubatch-size` peaks at 512 and loses on both sides.** Below it the SMs are
not saturated (256 → 1046, 384 → 1068); at the top end 2048 is 10 % off the
peak (1077). The middle is a shallow basin: 768–1792 sit at 1130–1170.
- **The curve is deterministic, not noisy.** 512 came out 1192.88 / 1193.23 /
1193.28 / 1193.88 across four different batch values, 1024 came out 1151.77 /
1151.91 / 1151.94, and 2048 came out 1077.38 / 1077.54 / 1077.59. Three
independent batch columns agreeing to 0.4 tok/s is what makes the 2.4 % gap
between 512 and 1536 real rather than run-to-run drift.
- **The logical `--batch-size` is irrelevant here.** At a fixed ubatch, 2048,
4096, 6144 and 8192 land within 2 % of each other — and use *identical* VRAM,
because the compute buffers are sized by the ubatch alone.
- **Decode does not move at all.** 15.88 tok/s at 64k depth on every cell and
17.80 at depth 0, to ±0.05. Whatever this knob does, it does it to prefill.
## What it costs at the deployed shape
Prefill rates do not decide a deployment; VRAM does. The same shape
re-measured at 262144 over two slots (load, poll `/health`, read `rocm-smi`,
kill — no requests):
| Split | ubatch | GPU 0 free | GPU 1 free | gap |
|---|---|---|---|---|
| 36,29 — deployed before | 1536 | 4.71 GiB | 4.91 GiB | 0.20 GiB |
| 36,29 | **512** | 6.01 GiB | 7.32 GiB | 1.31 GiB |
| **35,30 — deployed after** | **512** | **6.44 GiB** | **6.90 GiB** | **0.46 GiB** |
Two things follow.
**The ubatch move is free money.** 1536 → 512 gives back 1.30 GiB on ROCm0 and
2.41 GiB on ROCm1 — 3.71 GiB in total — and the mean throughput goes *up*, not
down. The asymmetry is the interesting part: ROCm1 pays nearly twice what ROCm0
does per unit of ubatch, because it holds the output layers and the drafter's
graphs.
**And it moves the balance, so the split had to follow.** Those uneven
give-backs left ROCm1 1.31 GiB lighter, and one layer is one `-ts` step: 36:29
became 35:30, which lands at 0.46 GiB — a *single DeltaNet layer's* worth of
asymmetry, i.e. the floor on how well any split can balance this stack (0.43 GiB
per DeltaNet layer, 1.43 per attention layer). The 0.20 GiB gap of the old
configuration is not reachable at the new batch; 0.46 GiB is.
## The decision
```ini
BATCH_SIZE=6144 # throughput-neutral and VRAM-neutral; left at the old value
UBATCH_SIZE=512 # the measured optimum: fastest AND cheapest
TENSOR_SPLIT=35,30 # rebalanced for the new ubatch, measured not interpolated
```
Verified live after the deploy: `--tensor-split 35,30 --ubatch-size 512`, two
slots of 131072, `6.44 / 6.90 GiB` free — matching the pre-deploy measurement to
0.01 GiB — a tool call and two concurrent requests answered, draft acceptance on
both slots (0.63 / 0.48 on slot 1, 0.39 on slot 0), and no ROCm errors.
## The split: re-measure it at the deployed shape
The batch and the tensor split are **coupled**: compute buffers are not
symmetric across the two cards, so any batch change moves the balance point.
Order of work — sweep batch with llama-bench (above), THEN re-measure the
split at the DEPLOYED shape (262144, two slots, drafter loaded) by loading
the real service with a per-config env override and reading `rocm-smi`. A
llama-bench number cannot decide the split: it has no `--parallel`, sizes
its own context as `n_prompt + n_gen + n_depth`, and cannot load the drafter.
The settle gate is mandatory (above); wait for BOTH cards under ~1 GiB AND
the port to be free (`ss -ltn | grep :7770` until gone) before the next
load, or the transient server dies with `couldn't bind HTTP server socket`.
Free bytes follow `card = weights_share + KV_share + constant + ubatch_term`,
where the ubatch term costs the drafter-carrying card roughly **1.85×** the
other (measured 2.41 GiB vs 1.30 GiB going 1536 → 512). Balance by
measurement only: one `-ts` step is one layer, worth ~0.43 GiB on a DeltaNet
boundary and ~1.43 GiB on an attention one, so the reachable minimum gap is
about one DeltaNet layer, never zero. Reference: 36,29 @1536 → 4.71/4.91;
36,29 @512 → 6.01/7.32; **35,30 @512 → 6.44/6.90 (the shape that deployed)**.
Re-fit both ubatch coefficients in `tests/frank_medium_unit.bats` at the new
deployed point and re-anchor the per-card constants; check neighbouring
splits still FAIL the balance gate or the seam has stopped discriminating
(keep the 1.5 GiB/card floor and the one-DeltaNet-layer slack). Deploy,
then verify live: flags in `/proc/<pid>/cmdline`, `/slots`, VRAM matching
the pre-deploy measurement, a tool call, two concurrent requests,
`draft acceptance` lines on BOTH slots, 0 ROCm errors.
**Per-card utilization (who is hot, and the split lever)** — a different
question from batch/prefill. Measure decode tok/s AND per-card
`rocm-smi --showuse` at 2–3 neighbouring splits in ONE session. Pitfalls
(each cost a re-run): run the transient unit as ROOT (a `pkill -9
llama-server` as dank auto-restarts the service and `systemctl` as dank
fails "Access denied"); warm up 3 throwaway requests before the timed one
(first 384-token decode after load measured 1.77 tok/s against a true ~32);
`rocm-smi --showmeminfo vram --csv` columns are `$2`=Total, `$3`=**Used** —
parsing the wrong column spins the settle gate the full 600 s; the util% is
sampled post-hoc (which card finished last), so treat it as "which card is
the pipeline tail", not a duty cycle.
**What the split does and does not buy (measured 2026-09-17, 393216 /
MTP n=3):** the split is NOT a decode-throughput lever — 34:31, 35:30, 36:29
all sit in the same ~32 tok/s band. The x1 card runs hot (~91–97 %) at
35:30 because the MTP drafter + output layers + heavier compute buffers are
pinned to the card holding the LAST layers — not because PCIe is the
bottleneck (~5.2 MB of boundary activation crosses per 512-token ubatch).
36:29 makes the x16 card the utilized worker (42 %/3 %) at no throughput
cost but spends headroom (0.45 GiB free, under the floor) — Frank's standing
preference is that the card with the FASTEST PCIe lane should be the more
utilized one; deploying it requires re-fitting the seam's slack/floor and
re-checking that 35:30 and 37:28 still fail so the gate keeps discriminating.
## Limitations
- **`llama-bench` cannot run the drafter, the slots or the deployed context.**
Its decode figure is the *drafterless* baseline (15.9 at 64k depth against the
service's ~44 with DFlash2 + ngram-mod), and it never exercises `--parallel` or
the 16 GiB KV of a real 262144 deployment. The throughput comparison between
batch values is sound; the absolute decode numbers are not the service's.
- **Five cells of the intended grid were not run** (batch 8192 above ubatch 512).
Batch 8192 at ubatch 512 matches the other batch columns to 0.4 tok/s, and the
sweep was stopped deliberately once the curve was established.
- **A totals model cannot see contiguity.** The earlier DSpark-era failure at
8192/2048 was a 2020 MiB pp buffer that found no room while every total looked
fine. Nothing here rules that in or out; it only shows there is no reason to
want a batch that large.
- **A measurement like this needs a settle gate.** Re-loading the model
immediately after a kill fails with `unable to allocate ROCm0 buffer` — the
previous instance's ~26 GiB is not returned instantly — which reads exactly
like a configuration that does not fit. Wait for `rocm-smi` to show both cards
near-empty before the next load.
## See Also
- [[frank-medium-llamacpp-dflash2]] — the deployment this tuned, and its split
history
- [[frank-zealot]] — the same sweep on a CUDA 3090, where a 16k prompt
at depth 0 made prefill climb all the way to ubatch 2048. Depth is the
difference: with no KV to attend over, per-pass overhead dominates and a
bigger micro-batch wins; at 64k of resident KV it does not.
- [[Dual-R9700-Cooperation]] — why the cards are layer-split at all
- [[Dankserver System Specs]] — hardware inventory
serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md
-732
View File
@@ -1,732 +0,0 @@
---
type: configuration
title: frank-medium llama.cpp Runbook (DFlash2 drafter)
description: The former ACTIVE frank-medium deployment since 2026-09-16 (model swapped 2026-09-19) — mainline llama.cpp on Dankserver serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the z-lab DFlash2 Q4_K_M drafter (the model's own MTP head stays in the file, one SPEC_TYPE change away), as ONE layer-split process across both Radeon AI PRO R9700 cards at a balanced 32:33 split, a 327680-position KV pool (163840 per conversation over two slots), f16 KV plus a 16 GiB host-RAM prompt cache, vision via the BF16 mmproj.
tags:
- frank-medium
- dankserver
- llama-cpp
- dflash2
- dspark
- qwen35
- gfx1201
- r9700
- layer-split
timestamp: 2026-09-20
---
> **SUPERSEDED 2026-09-20.** Replaced by [[frank-medium-radiance-mxfp4]]
> (radiance TP=2, measured faster on every axis). The layer-split shape below is
> the rollback target, and the llama.cpp measurements in this page stay valid for
> it — but nothing here is the deployed shape any more.
# frank-medium — llama.cpp (DFlash2 drafter)
The frank-medium deployment from 2026-09-16 to 2026-09-20 (model swapped to
orcarouter Q8_0 on 2026-09-19): **mainline llama.cpp**
serving `orcarouter/Qwen3.8-27B-Uncensored-GGUF` at `Q8_0`
with the **z-lab DFlash2** `Q4_K_M` block-diffusion drafter, as **one process across both
Radeon AI PRO R9700 cards**, layer-split at a balanced 32:33. The predecessor
model (ajgazin Swift-Qwen3.8-27B UD-Q8_K_XL, 35:30 skew, 393216 pool) is the
calibration history in the sizing tables below. DSpark was tried in the drafter's
place on 2026-09-16 and reverted — see *The DSpark excursion*.
It replaced the vllm-radiance MXFP4 replica pair + routing proxy
([[frank-medium-radiance-mxfp4]] holds that record, and the PCIe-atomics
evidence) the same day the pair went live — and was itself replaced by
radiance TP=2 on 2026-09-20, once the board swap made a collective legal.
During its window it was the box's **only** LLM runtime: the replicas, the
proxy and their unit files were retired.
## Why one process and layer split
The endpoint's whole history is one constraint: **the two cards could not take
part in an RCCL collective.** RCCL's kernels declare a hidden hostcall buffer,
and ROCr refuses to supply it to a device without PCIe atomics; on the previous
board the second R9700 hung off the B650 chipset (`AtomicOpsCap: Routing-`,
read-only), so every collective that touched it died with
`hipErrorIllegalState`. That ruled out tensor parallelism — and it ruled out
**vLLM's pipeline parallelism too**, whose stage handoff is a `send`/`recv` of
the hidden state on the device backend
(`vllm/distributed/parallel_state.py`), the very primitive that failed. The
full matrix is in [[Dual-R9700-Cooperation]].
**That gate cleared on 2026-09-20** with the ProArt B850-Creator swap: both
cards are on CPU root ports, every bridge reads `Routing+`, both endpoints read
`ReqEn+`. Nothing has dispatched a collective since, so tensor parallelism is
*unblocked, not proven* — and this service still runs the shape below, which
needs no collective either way.
llama.cpp's `--split-mode layer` needs **no collective at all**: layers are
distributed across the cards in order and the boundary activation is a plain
per-backend copy, once per forward pass. Measured on this host with the
predecessor runtime (llama-bench, pp4096 / tg128, f16 KV, flash-attn on):
| Split | pp4096 | tg128 |
|---|---:|---:|
| one R9700 (`-sm none`) | 1160.9 | 19.42 |
| **two cards, layer (`-sm layer`)** | **1991.2** | **18.76** |
| two cards, tensor (`-sm tensor`, no RCCL: butterfly fallback) | 271.8 | 26.26 |
Tensor split buys ~40 % decode and loses 7× prefill. The workload here is
prefill-heavy (agentic, long prompts, reasoning traces), so layer split is the
shape — prefill nearly doubles and one model serves from both cards.
## Hardware
| Property | Value |
|---|---|
| GPUs | 2x AMD Radeon AI PRO R9700 (DID 0x7551), gfx1201 (Navi 48, RDNA4) |
| VRAM | 34,208,743,424 B (31.86 GiB) each, as `rocm-smi` reports it |
| Link | PCIe 5.0 **x8** each on CPU GPP ports (x8/x8 bifurcation, ~31.5 GB/s per card), no XGMI; atomic routing enabled on both chains since the 2026-09-20 board swap — see [[Dankserver System Specs]] |
| iGPU | Raphael (gfx1036) enumerates as HIP device 2 — hidden via `HIP_VISIBLE_DEVICES=0,1` |
| CPU / RAM | Ryzen 7 7700X (8c/16t), 60 GiB RAM, 63 GiB swap |
## Runtime
Mainline llama.cpp at **`fb27a525`** (2026-09-16), built for **gfx1201 only** by
`scripts/build-llama-rdna4.sh` against a private ROCm `7.14.0a20260624`
toolchain (TheRock nightly) that is **bundled next to the binary** —
`RPATH=$ORIGIN/../lib`, no `LD_LIBRARY_PATH`, nothing from the host's ROCm 7.1
packages.
Two features pin that ref, and neither is in the ROCmFPX fork the endpoint used
earlier the same day (its tip, `c49ebdb`, predates both):
- **DFlash2** (`ggml-org/llama.cpp#27342`): grouped dynamic depthwise
convolution plus the candidate selector, detected from the draft GGUF
(`selector_top_k > 0`) and served through the ordinary
`--spec-type draft-dflash` path.
- **qwen35 MTP**: the `draft-mtp` implementation covers this architecture's
single trained head.
Build with:
```bash
bash scripts/build-llama-rdna4.sh # ~25 min on this box, JOBS=$(nproc)
```
> **Bundling trap, measured.** rocBLAS and hipBLASLt look for their device code
> at a fixed depth under the binary's lib dir (`../lib/rocblas/library`,
> `../lib/hipblaslt/library/<arch>`). `cp -r SRC DST/` with a missing `DST`
> creates `DST` as a copy of `SRC` and **flattens it one level**, and the
> failure is invisible at startup: the engine loads 29 GiB of weights, answers
> `/health`, and dies on the **first request** with
> `rocBLAS error: Cannot read .../lib/rocblas/library/TensileLibrary.dat: No
> such file or directory for GPU arch : gfx1201`. The build script now declares
> the destination and copies contents.
## Model and drafter
| File | Bytes | Role |
|---|---:|---|
| `Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf` | 31,457,990,112 | target, 29.29 GiB |
| `Qwen3.8-27B-DFlash2-Q8_0.gguf` | 2,056,414,816 | DFlash2 drafter, 1.92 GiB |
Fetched by `scripts/download-frank-medium-gguf.sh` (resumable, byte-verified —
a truncated 31 GB GGUF is otherwise a load-time crash).
The target is Unsloth Dynamic 3.0 (per-tensor types from Unsloth's own quant of
the same size, on Unsloth's importance matrix) of an abliterated Swift
fine-tune: orcarouter's single refusal direction projected out of 131 tensors,
vision tower included. Vision has been ON since 2026-09-17
(`--mmproj`); this is one of the fleet's two image endpoints.
**The file also carries the model's MTP head** (`blk.64.nextn.*`), and since
2026-09-17 that is the deployed drafter: `--spec-type draft-mtp` builds the
draft context against the main model's own weights (`shares_model`), so there
is no second copy of the model and no `--model-draft` argument — the cost is
one nextn layer's KV (4096 B/position) plus small buffers on the card that
holds the last layers (ROCm1). The DFlash2 sidecar stays on disk and in the
fetch script: `SPEC_TYPE=draft-dflash` (with the sidecar back on the command
line) brings the measured-faster chain back, at the price of ~1.1 GiB on ROCm1.
## Deployment
| File | Role |
|---|---|
| `scripts/frank-medium.env` | every tunable, with the sizing arithmetic in comments |
| `scripts/serve-frank-medium.sh` | env-driven llama-server launcher; no arguments |
| `scripts/frank-medium.service` | ROOT systemd unit, `User=dank`, `WantedBy=multi-user.target` |
```bash
./scripts/deploy.sh dankserver --service frank-medium
```
The handler verifies the runtime and payload exist (and refuses otherwise),
retires the three vLLM quadlets into `~/.AGENTTRASH/20260916-vllm-radiance-retire/`,
waits for **port 7770 to be released** (the retired proxy held it on the host
network), relinks and restarts the unit, and gates on
`http://127.0.0.1:7770/health`.
Deployed flags:
| Flag | Value | Why |
|---|---|---|
| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `32,33` | both cards, one model, no collective. Balanced (operator call 2026-09-19): 65 layers split as evenly as they can be, ROCm1 (the last-layers card that also carries the drafter and output layers) holding the extra one. The predecessor's 35:30 skew (2026-09-17, part 3) leaned work onto the x16 card because ROCm1's Gen4 x1 chipset path made it the 98 % bottleneck; the balanced point trades that lean for symmetry. Split granularity is one LAYER — 0.43 GiB on a DeltaNet boundary, 1.43 on an attention one |
| `--gpu-layers` / `--fit` | `999` / `off` | everything on the cards; auto-fit must not resize a context we set |
| `--ctx-size` / `--parallel` | `327680` / `2` | the pool over two slots of 163840; `--parallel` **divides** the context. 163840 is inside the checkpoint's trained 262144 window, so the whole pool is trained territory — no rope extrapolation (unlike the predecessor's 393216) |
| `--batch-size` / `--ubatch-size` | `6144` / `512` | measured: ubatch 512 is the prefill optimum *and* the cheapest point on the curve; the logical batch is throughput- and VRAM-neutral. See [[frank-medium-batch-ubatch]] |
| `--ctx-checkpoints` / `--cache-ram` | `0` / `16384` | no per-slot KV snapshots (the build's default is **32** — this is a deliberate deviation, not an inherited default); the RAM-side prompt cache is capped at 16 GiB, which is what lets a full 163840-token slot state be cached at all (2026-09-18, see below) |
| `--flash-attn` / cache types | `on` / `f16` K and V | an agentic checkpoint is the wrong place to spend KV precision |
| `--spec-type` / `--spec-draft-n-max` | `draft-dflash` / `7` | the z-lab DFlash2 `Q4_K_M` sidecar (1.06 GiB), the measured throughput winner (43.5 tok/s vs the model's own MTP head at 32.4); n_max 7 is the DFlash2 optimum. The target file also carries its own MTP head (`nextn_predict_layers=1`), so `SPEC_TYPE=draft-mtp` is one change away — the drafter file stays on disk |
| `--spec-draft-device` | `ROCm1` | the drafter rides the card that ends up holding the target's last layers |
| `--temp` / `--top-p` / `--top-k` | `0.7` / `0.95` / `20` | the values the vLLM predecessor served on this endpoint |
| `--reasoning-format` / `--reasoning-budget` | `deepseek` / `-1` | trace into `reasoning_content`, unlimited |
| `--mmproj` | `mmproj-Qwen3.8-27B-Uncensored-f16.gguf` (beside the target) | vision enabled; this build has NO autoload, so the flag is explicit and the serve script refuses to start without the file |
## The DSpark excursion (2026-09-16, tried and reverted)
The endpoint went live on **z-lab DFlash2**, was switched to
**magnitudedev/Qwen3.8-27B-DSpark** for a few hours the same day, and was put
back. Both are 5-layer block-diffusion sidecars, and their differences are what
decided it:
| | DFlash2 (deployed) | DSpark (tried, reverted) |
|---|---|---|
| file | 2,056,414,816 B — 1.92 GiB | 1,455,376,576 B — 1.36 GiB |
| draft depth | `block_size 8` -> `n_max 7` | `block_size 7` -> `n_max 6` (llama.cpp asserts `n_max <= block_size - 1`) |
| attention | `sliding_window 2048`, `causal false`, candidate selector (rank 256, top-k 16) | **no sliding window**; a Markov head in place of the selector |
| draft KV at 262144 | ~40 MiB — the window bounds it | **5.0 GiB** — the whole context |
> **A fetch-script bug this threw up.** The DSpark file's size was recorded as
> 1,359,284,737 when it was fetched — 96 MB short of the Hub's 1,455,376,576 —
> and that wrong number became the fetch script's declared size, so
> `--verify-only` failed on a payload that was in fact byte-correct. Both
> drafters' sha256 have since been checked against the Hub's LFS oids and match.
**The draft context is sized from the target's.** `cparams.n_ctx =
llama_n_ctx(ctx_tgt)` in `common/speculative.cpp`, with no flag to cap it
independently, so a drafter that declares no sliding window caches the entire
target window on whichever device `--spec-draft-device` names. DSpark's KV is
5 layers x 2 x 8 kv heads x 128 dim x 2 B = 20 KiB per position.
That cost has to be paid somewhere, and the failure was instructive: at a 1:1
split the model **loaded fine, answered `/health`, and then died on the first
draft decode** with `ROCm error: out of memory` inside
`common_speculative_impl_draft_dflash::process` — ROCm1 sat at 33.9 GB of
34.2 GB. A unit that crash-loops with an allocation failure *after* a
successful load is a sizing failure, not a broken drafter.
Read `sliding_window` out of a drafter's GGUF header before sizing a
spec-decode deployment — 40 lines of `struct` parsing is enough, and the
flag-level configuration looks identical either way.
### The split moved twice: once for the batch, once for the drafter
**Under DSpark.** Tripling the batch (2048/512 -> 6144/1536) made the two cards
diverge: at 262144 the bigger `--ubatch-size` cost **ROCm1 ~3.3 GiB against
ROCm0's ~1.7 GiB** — ROCm1 holds the output layers *and* the drafter's graphs —
which left the draft-carrying card at 0.08 GiB free against ROCm0's 3.78 under
the 38:27 split that had been balanced at the smaller batch:
| `-ts` | GPU 0 free | GPU 1 free |
|---|---:|---:|
| 38,27 | 3.78 GiB | 0.08 GiB |
| 41,24 | 1.47 GiB | 2.38 GiB |
| 40,25 | 1.88 GiB | 1.88 GiB |
**Then DSpark came off**, which frees ~4.3 GiB on ROCm1 (5.0 GiB of draft KV
out, 0.65 GiB of heavier DFlash2 weights in) and moves the balance the other
way. Re-measured at the same batch, the same two slots and the same window:
| `-ts` | GPU 0 free | GPU 1 free | gap |
|---|---:|---:|---:|
| 39,26 | 2.42 GiB | 7.20 GiB | 4.78 |
| 38,27 | 3.85 GiB | 5.77 GiB | 1.92 |
| 37,28 | 4.28 GiB | 5.34 GiB | 1.06 |
| **36,29 — deployed** | **4.71 GiB** | **4.91 GiB** | **0.20** |
| 35,30 | 5.14 GiB | 4.48 GiB | 0.66 |
| 34,31 | 6.57 GiB | 3.05 GiB | 3.52 |
**One `-ts` step is one layer, and this stack's layers are not the same size.**
qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is worth
**~0.43 GiB** on a DeltaNet boundary and **~1.43 GiB** on a full-attention one
(0.45 GiB of weights plus ~1.0 GiB of its own f16 KV at 262144). Both fell out
of that table — 37,28 -> 38,27 moves a DeltaNet layer, 38,27 -> 39,26 an
attention layer. Balance by measurement, never by interpolating.
**Then the batch moved, and the split followed it** (2026-09-16, the same day
— see [[frank-medium-batch-ubatch]]). 36:29 was the balance at ubatch **1536**;
the sweep showed 512 is both faster and 3.71 GiB cheaper, and — because the
compute buffers cost ROCm1 nearly twice what they cost ROCm0 — shrinking them
handed ROCm1 1.31 GiB more than it handed ROCm0:
| `-ts` | ubatch | GPU 0 free | GPU 1 free | gap |
|---|---:|---:|---:|---:|
| 36,29 | 1536 | 4.71 GiB | 4.91 GiB | 0.20 |
| 36,29 | 512 | 6.01 GiB | 7.32 GiB | 1.31 |
| **35,30 — deployed** | **512** | **6.44 GiB** | **6.90 GiB** | **0.46** |
35:30's 0.46 GiB gap is one DeltaNet layer's worth, which is the floor here: no
split can be tuned finer than a layer, so the 0.20 GiB of the old configuration
is not reachable at this batch. It clears the operator's 1.5 GiB-per-card floor
by ~5 GiB, so no headroom is bought out of `CTX_SIZE` and the model's full
262144-token window stays.
A batch of 8192/2048 failed here under DSpark — its 2020 MiB pp buffer could not
be allocated on ROCm1, which had 0.08 GiB free. That was a *contiguity* failure,
not a total-bytes one, and the sweep has since removed the reason to care: with
the batch geometry measured, there is nothing to gain by making either knob
larger, so the question is closed rather than pending.
> **A fault to watch.** A 39:26 split, at 1x batch and one slot, died on its
> eighth request: the amdgpu driver logged SDMA0 UTCL2 page faults and the
> engine surfaced `ROCm error: an illegal memory access was encountered` ->
> `ggml_abort` — a fault in the cross-card boundary copy, which is this layer
> split's hot path. It has not reproduced (36:29 served the whole verification
> battery and the e2e gate, and 35:30 at ubatch 512 is the deployed shape now;
> 39:26 itself was only *loaded* at the deployed batch, not exercised), and it
> is **not** an OOM: no amount of headroom would prevent it.
## 2026-09-17: the pool grows to 327680, the split moves to 36:29, vision comes on
The window grew from 262144 to a **327680-position pool** (163840 per
conversation over the two slots), the split moved one layer to ROCm0
(35:30 → 36:29), and the **BF16 mmproj** (888 MiB) came on.
**Why 36:29, not the balance.** At this context the fit model puts the balance
point at 34:31 (0.11 GiB gap). The operator moved a layer to ROCm0 to buy
headroom on ROCm1 — the card that carries the drafter, its KV, the output
layers and the heavier compute buffers — accepting a 3.15 GiB gap by
construction:
| Card | Fit | Measured at load | Headroom |
|---|---:|---:|---:|
| ROCm0 (36 layers + mmproj) | 29.19 GiB | 28.40 GiB | 3.46 GiB |
| ROCm1 (29 layers + drafter) | 26.04 GiB | 26.66 GiB | 5.20 GiB |
Both clear the 1.5 GiB floor with room; the fit lands within 0.8 GiB on both
cards, which is the accuracy it has always had (at the 35:30 point it
reproduced ROCm1 to 0.00 and ROCm0 to 0.86). The seam's slack is the 3.15 GiB
gap rounded up, with the trade written in its comment.
**The rope caveat.** The GGUF carries no `context_length` and no rope-scaling
keys (freq_base 1e7, `dimension_sections [11,11,10,0]`); the checkpoint's
trained window is 262144. Positions beyond that — the top 65k of the pool —
extrapolate on the base rope. It runs, and 1.25x is a modest ask of a rope,
but it is untrained territory, and the seam's ctx cap encodes 327680 with
that note.
**Vision: this build has no mmproj autoload.** The first 327680/36:29 deploy
came up healthy and *text-only*: image requests answered 500 "image input is
not supported — hint: … provide the mmproj" and the load log had no mmproj
line, so `llama-server` fb27a525 does not pick up `mmproj*.gguf` from the
model directory. The serve script now passes `--mmproj` explicitly (the
frank-zealot pattern), refuses to start without the file when vision is on, and
the deploy handler's preflight checks for it. Verified end-to-end after the
fix: a generated 64x32 test image was described from its pixels — with
thinking on the model reasoned about "the three horizontal bars" (its reading
of the crude glyph) and with thinking off answered the letter question in one
token.
The e2e crawl spec flaked once on the first 327680 deploy (1 failed / 31
passed) and passed 32/32 on the redeploy; the service itself was healthy
throughout (health gate green, `NRestarts=0`).
## 2026-09-17 (part 2): the drafter switches to MTP, the pool grows to 393216
The operator asked for **2x196k contexts**, which the DFlash2 sidecar's
~1.1 GiB on ROCm1 was eating. The drafter switched to the model's own
**MTP head** (`SPEC_TYPE=draft-mtp`, `n_max 3` — the measured MTP winner from
the same-day A/B: 32.4 tok/s at n_max 3 vs 26.7 at n_max 7, against DFlash2's
43.5), the pool grew 327680 → **393216** (196608 per conversation), and the
split moved one layer back to **34:31**, the balance point at the new pool.
| Card | Fit | Measured at load | Headroom |
|---|---:|---:|---:|
| ROCm0 (34 layers + mmproj) | 29.77 GiB | 28.91 GiB | 2.95 GiB |
| ROCm1 (31 layers + MTP buffers) | 28.32 GiB | 30.78 GiB | 1.08 GiB |
The fit predicted 28.32 on ROCm1; the load came in **2.46 GiB hotter** than
predicted. The MTP charge is one nextn layer's KV (4096 B/position — the
draft context is sized from the target, same law as the DFlash2 one) plus
draft buffers the fit only estimated. The seam was recalibrated to the
measured value: the MTP buffer constant is now 2,901,231,913 B (the 4.20 GiB
total MTP charge on card 1), and the floor dropped 1.5 → **1.0 GiB** for this
shape — ROCm1's 1.08 GiB of headroom is real. 35:30 failed card 0's floor
only at the OLD 1.5 GiB bar (it clears 1.0 with 1.27 GiB to spare); 36:29
fails both gates. Under the recalibrated 1.0 GiB floor 35:30 became
reachable — and the split went there the same day (part 3).
The rope caveat deepens: the top 131k of a conversation (196608 vs the
trained 262144… i.e. 393216/2 = 196608 per slot, so the top 131,072 positions
of a FULL conversation) extrapolates on the base rope. 1.5x is a bigger ask
than 1.25x; the seam's ctx cap encodes 393216 with that note.
Verified after deploy: `adding speculative implementation 'draft-mtp'` with
`shares_model` in the load log, no `--spec-draft-model` flag, `--mmproj`
present; 207 drafts / 108 accepted in a text run (52 % acceptance, mean len
~1.7); vision, tools and `reasoning_effort` all pass; e2e 32/32; `NRestarts=0`.
## 2026-09-17 (part 3): the PCIe topology, and the split leans to 35:30
`lspci -vv` on dankserver answers why the split now leans toward ROCm0:
the two cards do not sit on equal roads.
| Card | Path | Bottleneck link |
|---|---|---|
| ROCm0 `03:00.0` | CPU root port `00:01.1` (Gen4 x16) -> switch -> card | Gen4 **x16** (~31.5 GB/s) |
| ROCm1 `0b:00.0` | CPU root port `00:02.1` (Gen4 x4) -> chipset -> `06:09.0` -> `09:00.0` -> card | Gen4 **x1** (~1.97 GB/s) |
With the MTP drafter, the output layers and the heavier compute buffers
all pinned to the last-layers card, live `rocm-smi` under traffic read
**98% busy on the x1 card vs 40% on the x16 card**. The boundary crossing
itself is cheap (~5.2 MB of activation per 512-token ubatch rides the
wire — a few ms at 1.97 GB/s against a ~430 ms ubatch), but the
saturated card should not be the one behind the narrowest link. The
operator moved one layer to ROCm0: **34:31 -> 35:30**. The moved layer
is 34 — the 4k+2 position of block 8, a DeltaNet — which is what keeps
the shift cheap (~0.43 GiB).
Fit at the new shape: card 0 30.59 GiB (free 1.27), card 1 30.01 GiB
(free 1.85), gap 0.58 — inside the 1.0 GiB floor and the 1.483 GiB
slack. The next step, 36:29, crosses the full-attention layer at 35 and
fails BOTH gates (card 0 free 0.45 GiB, gap 2.22 GiB), so the
layer-toward-ROCm0 lever is spent: further rebalancing means a ctx cut,
not a layer. The seam's balance test re-derives all of this from the env
file on every run.
Verified after deploy: `rocm-smi` reads **30.84 GiB used on ROCm0
(headroom 1.02) and 28.86 GiB on ROCm1 (headroom 3.00)** — the shift
landed 1.80 / 1.97 GiB, more than the fit's 0.58 GiB gap predicted, and
the x1 card's headroom tripled. Utilization under live traffic: 60% on
the x16 card (was 40%), 98% on the x1 card — the drafter, the output
layers and the heavier compute buffers are structurally pinned to the
last-layers card, so one DeltaNet layer was all the split lever had left
to give. A chat completion answers in 1.4 s end-to-end; MTP acceptance
runs 0.51–0.91 (mean len 2.7–3.7) on the live traffic.
## 2026-09-18: the prompt cache doubles to 16 GiB
`--cache-ram` is the one cache tunable on this service that is **not VRAM**: it
caps the total bytes of saved prompt states in host RAM (`server_prompt_cache`,
`tools/server/server-task.cpp`). `update()` evicts oldest-first past the cap,
and `alloc()` **refuses outright** any single state larger than the whole cap —
with one `WRN` line and nothing else:
```
- prompt state size 11388.990 MiB exceeds cache size limit 8192.000 MiB, skipping
```
That line appeared **137 times** in the two days from 2026-09-16: at 8192 MiB
every conversation past ~170k tokens lost its state on the next slot
reassignment and had to be re-prefilled in full. It matters more here than the
cap alone suggests — this is a hybrid model (16 x 3 DeltaNet + 1 gated
attention), so a slot's recurrent cell sits at the end of its own prompt and
`--ctx-checkpoints 0` leaves no checkpoint to roll back to; llama.cpp then
takes the "forcing full prompt re-processing due to lack of cache data" path.
The RAM-side cache is the only mechanism that can carry a state *across* slots,
and a refused state is simply gone. The cap is a token budget in disguise,
because a state is the serialized KV of a whole sequence plus its
constant-size DeltaNet state plus the MTP draft context's own KV:
| Component | Per token |
|---|---|
| 16 full-attention layers x 4 kv heads x 256 dim x 2 B x 2 (K,V) | 64 KiB |
| MTP draft context (one nextn layer, same arithmetic) | 4096 B |
| **state law** | **68 KiB** |
| Cap | Tokens storable | Fits one 196608-token slot (~12.8 GiB)? |
|---|---|---|
| 8192 MiB (until 2026-09-18) | ~120k | no — refused, then re-prefilled |
| **16384 MiB (deployed)** | ~240k | **yes**, with ~3 GiB of budget spare |
The cost is host RAM, allocated on demand and never pinned, so no VRAM figure
above moves. dankserver has 62 GiB beside the media-stack co-tenants, which is
why the cache is bounded rather than set to `-1` (unlimited): the seam test in
`tests/frank_medium_unit.bats` gates both ends — the cap must clear one full
slot's state, and it may not claim more than half the host's RAM.
## Verified serving behaviour (2026-09-16)
- Load: `model loaded` ~25 s after start with a warm page cache, then
`listening on http://0.0.0.0:7770`. The drafter adds ~2.5 s.
- Slot shape: `n_slots = 2, n_ctx_slot = 196608`; `/props` reports
`modalities: {vision: true}` and the sampler defaults above.
- The drafter is engaged — the init lines are the proof, not the flags.
Current (MTP, since 2026-09-17 part 2):
```
common_speculative_impl_draft_mtp: adding speculative implementation 'draft-mtp'
common_speculative_impl_draft_mtp: - n_max=3, n_min=0, p_min=0.00
... (draft context shares the main model's weights: `shares_model`)
```
Under the DFlash2 era (2026-09-16 through 2026-09-17 part 1) the block was:
```
common_speculative_init_result: loading draft model '/home/dank/models/Qwen3.8-27B-DFlash2-GGUF/Qwen3.8-27B-DFlash2-Q8_0.gguf'
common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dflash'
common_speculative_impl_draft_dflash: - n_max=7, n_min=0, p_min=0.00
common_speculative_impl_draft_dflash: - block_size=8, mask_token_id=248070, n_extract=5, sample_from_anchor=true
```
`block_size=8` and `n_extract=5` are DFlash2's own metadata; the candidate
selector is what distinguishes it from DFlash 1. DSpark logs under the same
`draft_dflash` symbol with `block_size=7` and a Markov head, so read the
numbers, not the implementation name.
- Both cards hold the model. The measurement below is the **34:31-era**
record of the part 2 cutover; the deployed 35:30 shape is measured in
the part 3 section (ROCm0 30.84 GiB / ROCm1 28.86 GiB, headroom
1.02 / 3.00 GiB, both above the 1.0 GiB floor):
at 393216 with the deployed batch and the 34:31 split, `rocm-smi`
reported **28.91 GiB used on ROCm0 and 30.78 GiB on ROCm1** of 31.86
GiB each — the fit predicted 29.77 / 28.32, so ROCm1 ran 2.46 GiB
hotter than predicted and the seam's MTP constant was recalibrated to
the measured value (see the 2026-09-17 part 2 section). ROCm1's
1.08 GiB of headroom cleared the 1.0 GiB floor that shape required.
- Drafting is live on **both slots** (the 2026-09-16 DFlash2-era numbers below
are the record; MTP-era acceptance is in the part 2 section), which is what
a 2-slot deployment has to
prove: `slot print_timing` reported `draft acceptance = 0.62857 (44 accepted /
70 generated), mean len = 5.40` and `0.48214 (27 / 56)` on slot 1, and
`0.38776 (57 / 147), mean len = 3.71` on slot 0, from one concurrent pair.
Acceptance moves with the text; the point is that neither slot is idle.
- **140,069 prompt tokens prefilled in 134 s** (1043 tok/s) was measured
through a *single* slot, at the previous configuration: the raised window is
real rather than declared, and it is 22 % beyond the 114688 the vLLM
predecessor could offer. The window is now split over two slots.
- **Nothing is replicated.** The loader does not print a per-device ledger at
this verbosity, so this is measured instead of read: raising CTX_SIZE from
131072 to 262144 added **4.83 GB to GPU 0 and 4.84 GB to GPU 1** — both cards
carry the KV for their own attention layers, about half each — and neither
card's *total* usage (24.94 / 29.42 GB at 262144) could hold the 31.46 GB
file, so no card carries a full copy of the weights. Compare the shape this
replaced: two vLLM replicas, i.e. two complete copies of the model, one per
card, which is exactly why a 29.8 GB FP8 checkpoint could not be served there
at any usable context.
### Measured throughput (2026-09-16, deployed service, `temperature 0`)
Same filler corpus for every shape, so these are shapes rather than a
benchmark: a large prompt with a tiny budget isolates prefill, a small prompt
with a long budget isolates decode. `draft`/`acc` are per request; acceptance
moves with the text (filler prose accepts far worse than code or a real agent
loop).
| Shape | Prefill | Decode | Acceptance |
|---|---|---|---|
| 4k prompt / 32 out | 3994 tok in 3.25 s — 1230 tok/s | 59.8 tok/s | 71 % |
| 1k prompt / 512 out | 998 tok in 1.41 s — 707 tok/s | **43.8 tok/s** | 41 % |
| 16k prompt / 16 out | 12,528 tok in 8.94 s — **1402 tok/s** | — | 100 % (13/13) |
| 64k prompt / 16 out | 48,508 tok in 39.8 s — 1219 tok/s | — | 53 % |
| 140k prompt / 16 out | 140,069 tok in 134.3 s — 1043 tok/s | — | — |
| same 1k prompt again | **4 tokens prefilled** | 43.8 tok/s | 41 % |
The last row is the prefix cache doing its job: the second request of the same
conversation re-prefilled 4 of 998 tokens. Prefill holds ~1.0-1.4k tok/s from
4k to 140k, and decode sits at ~44 tok/s on a long generation — against the
vLLM predecessor's 42.8 tok/s single-stream (two replicas) and the ROCmFPX
llama.cpp measured here at 35.7-46.2 tok/s. One process, one model, both cards,
no proxy.
### What the drafter is worth — and why MTP is not the alternative
Measured 2026-09-16 with `scripts/bench-spec-ab.py` on the **DFlash2** drafter —
the one deployed now: four configurations of the same binary, identical prompts
at `temperature 0`, unit stopped between runs (one host, one copy of a 29 GiB
model). 1k-token prompt, 512-token budget. These are the numbers that selected a
trained multi-token drafter over the model's own MTP head.
| Configuration | Decode | Drafting | VRAM (card 1) |
|---|---:|---|---:|
| `none` — autoregressive | 17.7 tok/s | — | 26.07 GB |
| `draft-mtp` n_max 3 (the model's own head) | 32.4 tok/s | 58 % of 72 | 29.35 GB |
| `draft-mtp` n_max 7 | 26.7 tok/s | 33 % of 147 | 29.65 GB |
| **`draft-dflash` n_max 7 — deployed** | **43.5 tok/s** | 41 % of 119 | 29.42 GB |
- **DFlash2 is worth 2.46× over autoregressive and 34 % over the best MTP.**
Deeper MTP drafting costs more than it returns: at depth 7 only a third of
the proposals survive, at depth 3 more than half do.
- **All four produced identical greedy text** (`2, 3, 5, 7, 11`): speculation
buys speed, never a different answer. The harness warns loudly if a future
configuration ever breaks that.
- **MTP is not the cheaper option.** Selecting it loads `blk.64.nextn.*` from
the target and adds ~3.3 GB to the card that holds it, against 2.06 GB for
the drafter's file — which is also why the load log says
`unused tensor blk.64.nextn... ignoring` under `draft-dflash`: correct
behaviour, not a truncated download.
- **They cannot be chained.** `--spec-type draft-mtp,draft-dflash` dies at load
with `context type MTP requested but model doesn't contain MTP layers`. A
chain has ONE draft context, and MTP builds against whatever model that
context holds — here the DFlash2 file. MTP's head is loaded only when MTP *is*
the draft model (`creating MTP draft context against the target model ...`).
That is structural, not a tuning question.
### Chaining the draftless n-gram matcher (deployed)
`ngram-mod` **can** ride beside a context-backed drafter, and it is enabled.
The reason is in `common/speculative.cpp`: the n-gram types are added to the config vector
unconditionally while every model-backed drafter is gated on
`ctx_dft != nullptr` — so n-gram needs no draft context and cannot collide with
the drafter over one. (MTP is the opposite: it *is* context-backed, which is why
`draft-mtp,draft-dflash` cannot load.) The vector's fixed order — ngram types
first, then the drafters — also means CLI order is irrelevant.
The measurements below are DFlash2's — the deployed configuration. (During the
DSpark excursion the chain was `draft-dspark,ngram-mod`; the matcher's behaviour
is a property of the matcher, not of the drafter beside it.)
Measured with `scripts/bench-spec-ab.py`, decode tok/s, three shapes that a
proposer can and cannot match:
| Configuration | prose 1k/512 | echo 1.5k/400 (copy 0.94) | structured/400 | identity |
|---|---:|---:|---:|---:|
| `draft-dflash@7` | 43.77 | 76.82 | 91.08 | 70.37 |
| `ngram-mod` alone | 17.72 *(0 drafts)* | 102.50 | 17.80 *(0 drafts)* | 17.77 |
| **`draft-dflash,ngram-mod@7`** | **43.65** | **200.58** | 91.22 | 70.40 |
- **On prompt-echo traffic the chain is 2.6× the drafter alone** (200.6 vs 76.8)
and twice `ngram-mod` by itself, at 100 % acceptance.
- **On prose it is indistinguishable from DFlash2 alone** — the same 119 drafts
and 49 accepts, the same 43.7 tok/s. That is the property that makes it safe
as a default: where the matcher cannot match, it costs nothing.
- `ngram-mod` alone is a *negative control*: zero drafts on prose and identity
(17.7 tok/s = the autoregressive baseline), 100 % acceptance on echo. Nothing
else could have proposed those tokens, which is how its drafts are identified
at all — see the trap below.
- VRAM is unchanged (29.42 GB on card 1 with or without it): the matcher is a
64 MiB host-RAM hash table.
- `structured` (25 records with unique ids) rewards no proposer: its lines
differ, so there is no long span to match. It is a negative control for
*self-repetition*, not evidence about it.
**The window matters, and shorter is not better.** The chain at three windows:
| `SPEC_NGRAM_N_MIN`/`N_MAX` | prose 1k/512 | echo 1.5k/400 |
|---|---:|---:|
| **48/64 — deployed** | **43.9** | **200.6** |
| 16/32 | 36.0 | 161.0 |
| 8/16 | 36.2 | 113.8 |
A short window makes the matcher propose on ordinary prose fragments, where the
proposals are rejected and the verification is still paid — the "ngram hurts"
result this fleet measured twice before on other payloads. At 48/64 the chain
keeps DFlash2's exact numbers on prose and wins on echo.
> **Operational trap: `ngram-mod` prints no startup banner in this build.**
> `journalctl -u frank-medium` shows `adding speculative implementation
> 'draft-dflash'` and nothing for the matcher — the only evidence it is running
> is behavioural (`draft_n` > 0 where text repeats) or the per-task line
> `slot print_timing: ... draft acceptance = 1.00000 (256 accepted / 256
> generated), mean len = 65.00`. Do not conclude from the load log that the
> chain is not engaged.
### The agent-loop shape (fleet harness, 2 families × 2 turns)
`scripts/bench-turn-latency.py http://127.0.0.1:7770 2 2 LAYERSPLIT` — two
agent loops re-sending a ~30k-token document concurrently:
| | turn 1 | turn 2 |
|---|---:|---:|
| family 1 | 24,799 ms | 30,697 ms |
| family 0 | 51,424 ms | 8,015 ms |
Read that against the prefill table, not as a latency regression: with **one**
slot the second family's first turn waited for the first (25 s of cold 30k
prefill + the wait ≈ 51 s), and the last number — 8.0 s — was a re-prefill the
cache could only partly serve. That was the 1-slot configuration. The
deployment now runs **two slots of 131072**, so a second agent loop has a slot
of its own instead of queueing, at the cost of halving the window each one
gets. `--parallel` splits the context and the KV between slots; it does not add
throughput, so the trade is concurrency against conversation length.
### Benchmarking this endpoint
`scripts/bench-batch-ubatch.sh` sweeps `--batch-size` against `--ubatch-size`
with `llama-bench` and records, per cell, the prefill rate, the decode rate and
each card's **peak** VRAM. Run it from a transient unit rather than a shell:
```bash
sudo systemd-run --unit=fm-bench --collect --no-block --property=Type=oneshot \
--property=LimitMEMLOCK=infinity --property=User=dank \
scripts/bench-batch-ubatch.sh
```
Properties of `llama-bench` shape that harness, and three of them have already
produced wrong numbers on this fleet:
- **It separates devices differently from `llama-server`.** `/` splits devices
*within one configuration*, `,` splits *sweep values* — so this service's
`--tensor-split 36,29` is `-ts 36/29` to llama-bench, and the comma form
silently benches each card on its own instead. Likewise `--device
ROCm0,ROCm1` → `-dev ROCm0/ROCm1`.
- **There is no `-c`.** The context is a consequence of the test shape:
`n_ctx = n_prompt + n_gen + n_depth`. No bench can therefore prove that a
configuration's activation buffers fit beside the *deployed* KV — that needs
the live unit at 327680 across two slots.
- **A test that cannot allocate calls `exit(1)`**, taking the whole process with
it, so the harness runs one invocation per cell and a failure costs one TSV
line instead of the rest of the sweep.
- **A shell's memlock limit is not the unit's.** The unit sets
`LimitMEMLOCK=infinity` for the ROCm staging buffers; an SSH shell here gets
8 MiB, which fails the load for a reason that has nothing to do with the
model being benched.
- **No `--spec-*` and no `--parallel`.** The drafter cannot take part, so a
`llama-bench` decode figure is the *drafterless* baseline — 15.9 tok/s at 64k
depth against the deployed 43.8 — and slots are never exercised. `-d <n>` is
what puts KV in front of the measurement at all: a `tg` instance runs with
`n_prompt = 0`, i.e. at ~zero depth unless `-d` says otherwise.
## Observability
The endpoint exports Prometheus metrics under the **`llamacpp:`** namespace
(`--metrics`), which is what the fleet's scrape and dashboard read — the
`vllm:` series the predecessor exported no longer exist:
| Need | Series |
|---|---|
| Throughput | `llamacpp:prompt_tokens_seconds`, `llamacpp:predicted_tokens_seconds` |
| Queue | `llamacpp:requests_processing`, `llamacpp:requests_deferred` |
| Cache reuse | `llamacpp:prompt_tokens_cached_total` vs `llamacpp:prompt_tokens_total` |
| **Speculative decoding is live** | `llamacpp:spec_decode_num_drafts_total` and `llamacpp:spec_decode_num_accepted_tokens_total` (their ratio is the acceptance rate) |
`spec_decode_num_drafts_total` staying at 0 is the one-line answer to "is the
drafter actually running": the flags can all be right while the draft model
failed to load.
## Client contract
OpenAI-compatible `/v1` on **7770**, alias **`frank-medium`**, streaming and
`/health`, `/props`, `/metrics` as llama.cpp provides them. Reasoning traces
are projected into `reasoning_content`. A request that sends no sampler fields
gets the server defaults above; a client that sends its own wins. Images ride
the BF16 projector — `/props` reports `modalities: {vision: true, video: true,
audio: false}` (read live 2026-09-18), and there is no audio anywhere in the
fleet.
## Troubleshooting
- **Loads, then dies on the first request with a rocBLAS Tensile error** — the
bundled device-library depth (see the bundling trap above); check
`build-rdna4-rocm714/lib/rocblas/library/` exists and is populated.
- **`hipErrorNoDevice` / a card missing** — `HIP_VISIBLE_DEVICES` must be
`0,1`; the iGPU enumerates as device 2 and must stay hidden.
- **Drafting not engaged** — read the journal for
`common_speculative_impl_draft_dflash` (DFlash2 and DSpark both log under that
symbol); a missing drafter file makes the serve script exit before
llama-server starts, and a drafter whose GGUF the build does not recognise
fails the load instead.
- **Why is `blk.64.*` "unused"?** That is the model's own MTP head and it is
correct for `draft-dflash`. It is not a truncated download:
`bash scripts/download-frank-medium-gguf.sh --verify-only` proves the bytes.
- **A rebuild segfaults at load** — the build is gfx1201-only on purpose; a
gfx1200 sibling binary links, loads the model and then segfaults on these
cards.
## See Also
- [[frank-medium-radiance-mxfp4]] — the retired vllm-radiance MXFP4 deployment
(replicas + proxy) and the PCIe-atomics evidence
- [[frank-medium-rocmfpx]] — the llama.cpp/ROCmFPX deployment earlier on
2026-09-15, whose `--split-mode layer` measurements this service reuses
- [[Dual-R9700-Cooperation]] — why the two cards cannot take part in a
collective, and every software escape that is closed
- [[frank-zealot]] — the fleet's other llama.cpp endpoint (RTX
3090, MTP, the image path)
- [[Dankserver System Specs]] — hardware inventory
- [[frank-medium-batch-ubatch]] — the batch/ubatch sweep that set the deployed
geometry and moved this split from 36:29 to 35:30: the ubatch optimum at 512,
why the logical batch is free, and the harness traps
serverAppConfigs-docs/LLM/frank-medium-model-swap.md
-186
View File
@@ -1,186 +0,0 @@
---
type: runbook
title: frank-medium Model Swap
description: RETIRED-era procedure (frank-medium is a retired endpoint; `:7770` on Dankserver is [[frank-archon]] today and the live swap law is [[Model Swap and Rollback]]) — how a checkpoint swap behind frank-medium worked: the streamlining toolchain (hf-manifest.py, hf-fetch.sh, swap-frank-medium.sh), the quantization-aware entrypoint contract, the quadlet edit, the deploy, the capability battery, and the rollback path.
tags:
- frank-medium
- dankserver
- model-swap
- vllm-radiance
- huggingface
- runbook
timestamp: 2026-09-20
---
# frank-medium Model Swap
**Historical.** frank-medium is a retired endpoint — `:7770` on Dankserver is
[[frank-archon]] today, and the live procedure is [[Model Swap and Rollback]].
This page records how a swap worked in that era: a four-step process whose design
goal (2026-09-20, after the MXFP4 → FP8 swap) was that a swap be **one command** —
generate a manifest, download it verified, point the quadlet at it, deploy. The
entrypoint was quantization-aware, so the only repo change a swap made was the
quadlet's four env vars.
## Why this exists
The 2026-09-20 MXFP4 → FP8 swap took most of a day, and the long pole was not
the GPU — it was the **download** (30.9 GB across 7 shards, hand-verified) and
the **serving-path decision** (which attention backend, which drafter, which
quantization flag). Both are now mechanical:
- The download is `hf-fetch.sh` — resumable, byte-count + sha256-verified,
idempotent (a re-run only fetches what is missing or corrupt).
- The serving path is the **quantization-aware entrypoint**
(`dankserver/frank-medium/radiance-entrypoint.sh`) — it reads
`quantization_config.quant_method` from the checkpoint's `config.json` and
picks the path. The swap operator never chooses it by hand.
## The toolchain
| Tool | Role |
|---|---|
| `scripts/hf-manifest.py` | Generate a fetch manifest from a live HF repo tree: `<filename> <size> <sha256-or-blank>` per line. LFS sha256s come from the resolve endpoint's `X-Linked-ETag`; non-LFS files get a blank hash (size-only check). |
| `scripts/hf-fetch.sh` | Fetch a manifest to a dir, resumable (`curl -C -`), verified (byte count + sha256), idempotent. Token out-of-band via a mode-600 `curl -K` file. |
| `scripts/swap-frank-medium.sh` | The orchestrator: generate manifest → download → detect quantization → edit the quadlet → deploy. |
The HF token is resolved on the **host** (dankserver) from `$HF_TOKEN`,
`~/.cache/huggingface/token`, or the llamaswap conf. Gated repos need a token
that has been granted access (the 2026-09-20 FP8 swap needed a one-time grant
for `orcarouter/Qwen3.8-27B-Uncensored-FP8`).
## The swap (one command)
```
scripts/swap-frank-medium.sh \
--repo <owner/repo> \
--target /home/dank/models/<name> \
--generate-manifest \
--apply \
--deploy
```
What it does, in order:
1. **Manifest** — `hf-manifest.py` lists the repo tree and writes
`scripts/frank-medium-<name>.manifest` (committed, so the payload is
reproducible). Pass `--manifest <path>` to reuse an existing one.
2. **Download** — stages the manifest + `hf-fetch.sh` on the host and fetches
to `--target`. Resumable: an interrupted swap re-runs and skips what is
already verified.
3. **Detect** — reads `quantization_config.quant_method` from the downloaded
`config.json` and maps it to the serving shape (below).
4. **Apply** — edits the quadlet's four env vars (`SNAP`, `SPEC_METHOD`,
`SPEC`, `KV_MEM`).
5. **Deploy** — `deploy.sh dankserver --service frank-medium --skip-e2e`
(pull, restart, health gate, up to 1800 s — the first start compiles
kernels).
Without `--apply` the script stops after the download + detection and prints
the quadlet lines to set, so you can eyeball them before committing.
## Quantization → serving path
The entrypoint (`radiance-entrypoint.sh`) reads `quant_method` and branches:
| `quant_method` | Path | Attention | Speculation | Drafter |
|---|---|---|---|---|
| `fp8` | plain FP8 | `ROCM_AITER_UNIFIED_ATTN` | **DFlash2 external drafter** (A/B winner) | `/models/Qwen3.8-27B-DFlash2-FP8` |
| `mxfp4` / `quark` / empty | MXFP4 | `R4D` (if `R4D_ATTN=1`) | DFlash2 external drafter | `/models/Qwen3.8-27B-DFlash2-FP8` |
The quadlet `SPEC_METHOD`/`SPEC` the swap script sets: **both paths use
`dflash/7`** (the DFlash2 drafter is quantization-agnostic — it proposes
tokens, the target verifies, so it works with any Qwen3.8-27B target). The
FP8 model also carries its own single-layer MTP head (22 tensors), so MTP
self-speculation (`mtp/8`) is available as an alternative — but A/B tested on
this checkpoint, **DFlash2 wins on decode** (100.0 vs 85.9 t/s geomean, 287.1
vs 179.7 t/s burst-4) despite a lower accept length (4.24 vs 5.66), because
the small separate drafter steps faster than the target's own MTP layer
applied autoregressively. The MXFP4 path additionally applies the
`radiance-vllm-mxfp4` patches at every container start.
**R4D allreduce is independent of the weight quantization.** The FP8 path
uses `ROCM_AITER_UNIFIED_ATTN` (the documented plain-FP8 default); R4D can be
tried later for a speed bump (`R4D_ATTN=1`), but the MXFP4 path is the one
tested with it.
## The quadlet contract
A swap edits exactly four env vars in
`dankserver/frank-medium/quadlet/frank-medium.container`:
- `SNAP=/models/<name>` — the container path (the host dir
`/home/dank/models/<name>` is mounted at `/models`).
- `SPEC_METHOD=mtp|dflash` — the speculative-decoding method.
- `SPEC=8|7` — the speculative depth.
- `KV_MEM=0` — the KV pool pin in MiB; `0` = auto-size (the documented
plain-FP8 default). After the first healthy start, recalibrate with
`calibrate-kv.sh` to use the extra VRAM on the 34.2 GiB cards.
The deploy handler (`deploy-remote.sh deploy_llm_medium`) reads `SNAP` and
`SPEC_METHOD` from the quadlet (the source of truth), maps `/models/` →
`/home/dank/models/`, and gates on the right assets: the drafter only for
`dflash`, `r4d.so` only for `mxfp4`/`quark`. So a swap that points the quadlet
at a new checkpoint is enough — the handler follows.
## Verify (capability battery)
After the deploy is green, run the battery (read-only — it does not stop the
service):
```
python3 scripts/bench-conversation.py http://192.168.0.235:7770 \
--model frank-medium --salt <name> --max-tokens 512 --burst 4
```
Then the capability checks (all must pass):
- `/health` → 200, `/v1/models` → `frank-medium`.
- A chat completion returns `content` **and** reasoning (vLLM emits the
`reasoning` field; the harness auto-detects it).
- MTP speculation: the boot log's `SpecDecoding metrics` line shows a mean
acceptance length > 1 (the 2026-09-20 FP8 swap measured 5.66/8, 66.2%
draft acceptance).
- Tool calls: a `tools` request returns `tool_calls`.
- Vision: an image request is answered (the FP8 checkpoint serves vision
despite `language_model_only: true` in `config.json`).
## Rollback
The previous checkpoint stays on disk (a swap never deletes it). To roll
back:
1. Point the quadlet at the old checkpoint (`SNAP`, `SPEC_METHOD`, `SPEC` —
the old values are in git history).
2. `deploy.sh dankserver --service frank-medium --skip-e2e`.
3. Re-run the capability battery.
The 2026-09-20 rollback target is the MXFP4 checkpoint at
`/home/dank/models/Qwen3.8-27B-Uncensored-MXFP4-awq` (19.37 GiB single
safetensors) with `SPEC_METHOD=dflash`, `SPEC=7`.
## 2026-09-20 swap record (MXFP4 → FP8)
- **Model**: `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4,
19.37 GiB) → `orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3 block-wise
128×128, dynamic activation quant, 30.9 GB / 7 shards).
- **Serving path**: DFlash2 external drafter → DFlash2 external drafter
(kept; `R4D` → `ROCM_AITER_UNIFIED_ATTN`). After the swap, MTP
self-speculation (`mtp/8`) was A/B tested against the DFlash2 drafter
(`dflash/7`) on the FP8 target.
- **DFlash2 vs MTP A/B** (the deployed shape is DFlash2):
- DFlash2: mean accept 4.24/7, geomean decode **100.0 t/s**, TTFT 0.19 s,
prefill 923.0 t/s, burst-4 **287.1 t/s**, KV pool 384,238 (1.47×).
- MTP: mean accept 5.66/8, geomean decode 85.9 t/s, TTFT 0.06 s, prefill
1096.9 t/s, burst-4 179.7 t/s, KV pool 461,540 (1.76×).
- **DFlash2 wins on decode (+16 %) and burst-4 (+60 %)**; MTP wins on TTFT
and prefill. Accept length is a proxy, not the metric — the small separate
drafter steps faster than the target's own MTP layer applied
autoregressively, and the faster step more than makes up the lower accept
length. FP8 + DFlash2 (100.0 t/s) also beats the original MXFP4 + DFlash2
(92.4 t/s) — the higher-precision FP8 target verifies more of the drafter's
proposals.
- **KV pool**: auto-sized to 384,238 tokens (1.47× full-window) — the
DFlash2 drafter's 2.1 GiB costs the pool vs MTP's 461,540 (1.76×); both
smaller than MXFP4's 943,581 (3.59×) because FP8 weights are bigger (30.9
vs 19.4 GiB).
serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md
-634
View File
@@ -1,634 +0,0 @@
---
type: configuration
title: frank-medium vllm-radiance MXFP6 Runbook
description: RETIRED era runbook (endpoint renamed frank-archon 2026-10-10; this shape is a row in [[frank-archon]]'s Eras table) — vllm-radiance serving hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 (ParoQuant MXFP6 W4A8, DFlash2 drafter, unquantized bf16 KV) as ONE tensor-parallel process (TP=2) on Dankserver's two AMD Radeon AI PRO R9700 (gfx1201). Cut to MXFP6 on 2026-09-21 (PR #51's launch6_p kernel, image 0.13.0-e060895); the MXFP4 checkpoint (just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq, the 2026-09-21 shape) is the on-disk rollback. Model-swap process: [[frank-medium-model-swap]]. The sections below describe the 2026-09-16 two-replica era.
tags:
- frank-medium
- dankserver
- vllm
- vllm-radiance
- mxfp6
- mxfp4
- gfx1201
- r9700
- rccl
- pcie-atomics
- quadlet
timestamp: 2026-09-21
---
# frank-medium vllm-radiance MXFP6 Runbook
**RETIRED era (2026-09-21 → 10-10; the endpoint is [[frank-archon]] today and this
shape is one row in its Eras table.)** The deployment was the `ggz14/vllm-radiance-mxfp4:0.13.0-e060895`
image (built from PR #51's head, tagged locally as
`localhost/radiance-vllm:0.13.0-e060895`) plus the `radiance-vllm-mxfp4`
patches at e060895, serving `hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6`
(ParoQuant MXFP6 W4A8, 25.16 GiB, 18 shards) as **ONE tensor-parallel
process across both Radeon AI PRO R9700 (gfx1201)**. It replaced the
mainline-llama.cpp layer-split deployment ([[frank-medium-llamacpp-dflash2]],
SUPERSEDED 2026-09-20); the same image and model had earlier run here as two
single-GPU replicas, which is the shape most of this page describes.
**Serving shape since 2026-09-21: ParoQuant MXFP6 with unquantized (bf16) KV.**
This endpoint cut to the MXFP6 checkpoint above (PR #51's `launch6_p` kernel,
`quant_method=paroquant_mxfp6`, bits=6, group_size=128, krot=8, rotation on,
e4m3fn scale) with `--kv-cache-dtype auto` (unquantized bf16 KV) and
`--gpu-memory-utilization 0.98`. The Swift variant is a finetune of
`ukisai/Swift-Qwen3.8-27b` (a different base from `Qwen/Qwen3.8-27B`), so the
DFlash2-FP8 drafter's hidden-state distribution is a mismatch — acceptance may
drop from the MXFP4 era's 4.24/7, but it is still a net win over no spec
decode. **The MXFP4 checkpoint is the on-disk rollback**: flip `SNAP` back to
`/models/Qwen3.8-27B-Uncensored-MXFP4-awq`, and the image
back to `localhost/radiance-vllm:0.9.3` (re-pull `docker.io/stilldeadcode/
vllm-radiance:0.9.3` if it is no longer on disk; the MXFP4 shape auto-sized to
486,464 tokens with bf16 KV and held the full 262,144 window).
**Deployed shape since 2026-09-20: one rootless Quadlet, `--tensor-parallel-size 2`.**
The ProArt B850-Creator WiFi NEO board swap put both R9700s on CPU GPP root
ports (`00:01.1 → 03:00.0` and `00:01.3 → 07:00.0`), both Gen5 x8, every
upstream bridge `AtomicOpsCap: Routing+` and both endpoints `ReqEn+` — the wall
documented in [Why TP=2 cannot run here](#why-tp2-cannot-run-here) is gone on this
board. The collective that actually runs is vLLM's R4D one-shot allreduce, which
does not use RCCL at all (TheRock RCCL has no gfx1201 device code; see
[[Dual-R9700-Cooperation]]).
## Deployed flags (2026-09-21)
From `dankserver/frank-medium/quadlet/frank-medium.container`:
| Setting | Value |
|---|---|
| Model | `hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6` (ParoQuant MXFP6 W4A8, 25.16 GiB, 18 shards; a finetune of `ukisai/Swift-Qwen3.8-27b`) |
| Speculative | `--speculative-config {"method":"dflash","model":"/models/Qwen3.8-27B-DFlash2-FP8","num_speculative_tokens":7}` (the z-lab DFlash2 external drafter; measured on the Swift finetune: mean acceptance length 7.00 / 85.7% on easy text, 5.08 / 58.3% on varied — the drafter survives the base-model change. A/B tested against MTP self-speculation — DFlash2 wins on decode, see below) |
| Parallelism | `--tensor-parallel-size 2` (`ROCR_VISIBLE_DEVICES=0,1`) |
| Context | `--max-model-len 262144` |
| Concurrency | `--max-num-seqs 8`, chunk 8192 |
| KV | auto = **unquantized bf16** (`--kv-cache-dtype auto`), **auto-sized — pool 425,047 tokens** (measured at 0.98 fill: 14.42 GiB per card available for KV, 1.62x the 262,144 window); rollback: `KV_DTYPE=fp8` |
| Prompt cache | `--enable-prefix-caching` |
| GPU fill | `--gpu-memory-utilization 0.98` (the 32 GB ceiling — 0.99 fails the startup check) |
| Parsers | `--enable-auto-tool-choice --tool-call-parser qwen3_coder`, `--reasoning-parser qwen3` |
| Sampler floor | `--override-sampling-effects` (temp 0.7 / top_p 0.95 / top_k 20) |
| Multimodal | `--limit-mm-per-prompt '{"image": 8}'` |
| Chat template | `qwen-sharp-v22.5.0.jinja` (peculiar-ragdoll Qwen-Sharp v22.5.0, repo-tracked at `dankserver/frank-medium/`, mounted at `/templates/`; terse system prompt by default — opt out per-request with `chat_template_kwargs {"terse": false}`; rollback: flip `CHAT_TEMPLATE` back to `/templates/qwen-sharp-v22.4.0.jinja`) |
**The KV pool is auto-sized, not pinned.** Production ships no `KV_MEM` pin
(`KV_MEM=0`): the pool comes up at whatever VRAM the weights + drafter leave.
MXFP6's weights are 25.16 GiB (~12.58 GiB per card at TP=2); at 0.98 fill
(31.22 GiB per card) that leaves **14.42 GiB per card for KV** — a measured
**425,047-token pool**, 1.62x the 262,144 window (boot 2026-09-22). The
DFlash2-FP8 drafter costs 2.1 GiB of VRAM on top.
**0.98 is the ceiling, not a preference**: the R9700 reports
31.86 GiB total but only 31.23 GiB free at device-init (the ROCm/HIP context +
NCCL + the radiance custom all-reduce buffers consume the 0.63 GiB gap), so a
`gpu_memory_utilization` above 0.98 (0.99 = 31.54 GiB) fails vLLM's startup
check before the weights even load. For reference the MXFP4 checkpoint (the
rollback) auto-sized to 486,464 tokens at the same 0.98 with bf16 KV (1.86x the
window); MXFP6's larger weights give a smaller pool, still comfortably above the
window. After the first healthy start, recalibrate with `calibrate-kv.sh`.
## DFlash2 vs MTP — A/B on the FP8 checkpoint (2026-09-20)
The FP8 checkpoint carries its own single-layer MTP head, so it can self-
speculate (`SPEC_METHOD=mtp`) or use the z-lab DFlash2 external drafter
(`SPEC_METHOD=dflash`, a separate quantization-agnostic 2.1 GiB
`DFlash2DraftModel`). Both were deployed and measured on the same FP8 target
(`scripts/bench-conversation.py`, 10 real prompts, `max_tokens=512`, greedy,
salted, `--burst 4`):
| Arm | Mean accept | Decode (geomean) | TTFT (mean) | Prefill (geomean) | Burst-4 aggregate | KV pool |
|---|---|---|---|---|---|---|
| **DFlash2** (depth 7) | 4.24/7 | **100.0 t/s** | 0.19 s | 923.0 t/s | **287.1 t/s** | 384,238 (1.47×) |
| MTP self-spec (depth 8) | 5.66/8 | 85.9 t/s | 0.06 s | 1096.9 t/s | 179.7 t/s | 461,540 (1.76×) |
**DFlash2 wins on decode (+16 %) and burst-4 (+60 %); MTP wins on TTFT and
prefill.** The counterintuitive part: DFlash2 accepts *fewer* tokens per step
(4.24 vs 5.66) yet decodes *faster*. Accept length is a proxy, not the metric
— throughput is `(accept + 1) / (draft time + verify time)`. The DFlash2
drafter is a small separate model that steps fast; the MTP head is the
target's own layer applied autoregressively 8×, so its draft is slower. The
faster step more than makes up the lower accept length. FP8 + DFlash2 (100.0
t/s) also beats the original MXFP4 + DFlash2 (92.4 t/s) — the higher-precision
FP8 target verifies more of the drafter's proposals. The drafter costs 2.1 GiB
of VRAM (the KV pool is 1.47× vs MTP's 1.76×), an acceptable trade for the
decode win. **DFlash2 is the deployed shape.**
## Prefix caching — measured granularity (2026-09-20)
`--enable-prefix-caching` is on, and radiance says so at boot: *"Prefix caching
in Mamba cache 'align' mode is currently enabled. Its support for Mamba layers
is experimental."* The attention block size is **1,648 tokens** (boot log:
"Setting attention block size to 1648 tokens …"), padded to the mamba/GDN page
size, and reuse happens at that granularity — not per token.
Measured on the deployed endpoint, not inferred:
| Probe | Cold | Warm | `vllm:prefix_cache_hits_total` delta |
|---|---|---|---|
| identical 13,021-token prompt | 2.30 s | 0.68 s (3.4×) | +9,888 of 13,021 queried (76 %) |
| identical 2,521-token prompt | 0.51 s | 0.51 s | +0 (under two blocks) |
Consequence: a shared prefix shorter than about two blocks (≈3.3k tokens)
re-prefills in full, so the benefit starts at ~3.3k tokens of shared prefix.
The API returns `prompt_tokens_details: null` in this build, so the
`vllm:prefix_cache_hits_total` / `vllm:prefix_cache_queries_total` counters
(both token-counted) are the only instrument — see [[Observability]].
## Battery results — why this shape won (2026-09-20)
`scripts/bench-conversation.py`, 10 real-workload prompts, `max_tokens=512`,
greedy, salted per run and per arm, service stopped between arms
(`scripts/bench-conversation.py --host http://127.0.0.1:7770 --burst 4`):
| Arm | Shape | Decode (geomean) | TTFT (mean) | Prefill (geomean) | Burst-4 aggregate |
|---|---|---|---|---|---|
| I | llama.cpp layer split 32:33 + DFlash2 Q4_K_M drafter | 45.7 t/s | 0.72 s | ~530 t/s | 71.7 t/s (26.7/stream) |
| L | llama.cpp `--split-mode tensor` 1:1, drafterless | 30.5 t/s | 0.71 s | — | 54.3 t/s |
| **R** | **vllm-radiance TP=2 + DFlash2-FP8** | **95.8 t/s** | **0.21 s** | **1493.6 t/s** | **259.4 t/s (65.6/stream)** |
Re-measured against the production unit after deploy: geomean decode 92.4 t/s,
mean TTFT 0.22 s, burst-4 aggregate 248.9 t/s — within noise of arm R.
Honest caveat: arm R carries MXFP4 (4-bit) where the llama arms carry Q8_0
(8-bit), so part of the win is quantization, not tensor parallelism alone. Arm
L exists to show that llama.cpp's own TP is not the answer here regardless —
it is slower than its own layer split, and cannot carry the drafter at all.
## Verified capabilities (2026-09-20, against the live endpoint)
- `/v1/models` → `id: frank-medium`; `/health` → 200.
- **Tool calls**: `finish_reason: tool_calls` via `qwen3_coder`. The harness
drives a bash tool through this endpoint end to end.
- **Reasoning**: `reasoning_effort` accepts `none`/`low`/`medium`/`high`
natively (vLLM's `ReasoningEffort`); thinking arrives in the `reasoning`
field, NOT llama.cpp's `reasoning_content`. `max_tokens` counts visible
content only; with no reasoning budget the model still spends it, so a tiny
budget returns `content: null` with `finish_reason: length`.
- **Vision works** despite `language_model_only: true` in `config.json` — the
Gemma4 merge path answers image requests (~83 prompt tokens for a small
synthetic), correctly identifying shape and colour on held-out images.
`models.yml` therefore lists this endpoint as `input: [text, image]`.
- Drafter is live: accept length 2.9–4.3 tokens per draft, acceptance ~0.60.
## Rollback
The llama.cpp layer-split trio is archived in `scripts/.AGENTTRASH/
20260920-llama-layersplit/` (`frank-medium.service`, `frank-medium.env`,
`serve-frank-medium.sh`) and the runtime + Q8_0 payload remain on disk:
1. `git revert` the cutover commit.
2. `deploy.sh dankserver --service frank-medium` (re-installs and re-enables the root unit).
3. Or by hand: restore the three files, move the unit symlink back into `/etc/systemd/system/`, `sudo systemctl daemon-reload && sudo systemctl unmask frank-medium && sudo systemctl start frank-medium`.
4. Gate: `curl -s http://192.168.0.235:7770/health` → 200.
## R4D vs RCCL
vLLM's R4D one-shot allreduce bypasses RCCL entirely because TheRock RCCL
builds lack gfx1201 device code (`invalid device function`). The Ubuntu
RCCL 2.27 / `librccl1` 7.1.0 build does load gfx1201 kernels — that is what
`scripts/build-llama-rdna4.sh` with `GGML_HIP_RCCL=ON` links
(`build-rdna4-rocm714-rccl/`) — so llama.cpp's tensor mode has a real
collective too. It is simply the slower shape.
> **The sections from here down describe the 2026-09-16 → 2026-09-19 era: two
> single-GPU replicas behind an adaptive-routing proxy, on the TUF GAMING
> B650-E.** The deployed shape is above; this is the evidence trail that made
> replicas necessary then and what the 2026-09-20 board swap changed.
## Architecture
```mermaid
graph LR
C[clients: Hermes, harness] --> P["frank-medium-proxy<br/>:7770 adaptive routing"]
P --> A["frank-medium-a<br/>R9700 #1, TP=1, :7771"]
P --> B["frank-medium-b<br/>R9700 #2, TP=1, :7772"]
```
Three rootless Quadlet units, all `WantedBy=default.target` + `Restart=on-failure`:
| Unit | Container | GPU | Port | Role |
|---|---|---|---|---|
| `frank-medium-a` | `frank-medium-a` | `HIP_VISIBLE_DEVICES=0` | 7771 | replica |
| `frank-medium-b` | `frank-medium-b` | `HIP_VISIBLE_DEVICES=1` | 7772 | replica |
| `frank-medium-proxy` | `frank-medium-proxy` | — | 7770 | routing |
The fleet contract is unchanged: one endpoint on :7770, served model name
`frank-medium`, OpenAI `/v1`, streaming included. The vLLM OpenAI API is
stateless per request, so no session affinity is needed for correctness. Routing
is described in [Routing](#routing).
Only `HIP_VISIBLE_DEVICES` is set per replica, never a matching
`ROCR_VISIBLE_DEVICES`: ROCR filters first and HIP then indexes into the
filtered list, so the single-valued pair (`ROCR=1,HIP=1`) leaves an empty
device set and the engine dies with `hipErrorNoDevice`.
`frank-medium.container` (TP=2, `KV_MEM=18563072000`, `MAXLEN=262144`) stays in
the repo as the target shape for a host whose second card sits on an
atomic-capable root port. `deploy-remote.sh` deliberately does not install it
and moves any stale copy into `~/.AGENTTRASH/`, because a TP=2 unit on this
host would restart forever without ever becoming healthy.
## Why TP=2 cannot run here
> **Resolved 2026-09-20.** Everything below is true of the **TUF GAMING
> B650-E WIFI**, whose second card sat behind chipset bridges advertising
> `AtomicOpsCap: Routing-`. The ProArt B850-Creator WiFi NEO swap moved both
> cards onto CPU GPP root ports (`00:01.1 → 03:00.0`, `00:01.3 → 07:00.0`, Gen5
> x8, every hop `Routing+`, both endpoints `ReqEn+`), and TP=2 now boots and
> serves. The measurement verdict against the alternatives is in
> [[Dual-R9700-Cooperation]].
### Symptom
Any two-rank collective dies at launch:
```
[FATAL ERROR]: HIP failure: 'the operation cannot be performed in the present state'
group.cc:337 -> group.cc:656 -> group.cc:801 -> enqueue.cc:3111
```
With the runtime's own API log (`AMD_LOG_LEVEL=3`) the failing call is explicit:
```
[ShaderName : ncclDevKernel_Generic_4(ncclDevKernelArgsStorage<4096ul>)]
[ :1:rocvirtual.cpp :4208: Pcie atomics not enabled, hostcall not supported]
[ :1:rocvirtual.cpp :4636: AQL dispatch failed!]
[ :3:hip_module.cpp :893 : hipLaunchKernel: Returned hipErrorIllegalState]
```
ROCclr refuses to hand the kernel a hostcall buffer when the device has no
PCIe atomics (`pcie_atomics_` is true for the device init, and ROCr passes
that buffer only when `pcie_atomics_` is true; otherwise it returns "not
supported" and the dispatch fails) — and the launch aborts.
`hipErrorIllegalState` is that refusal, not a driver fault.
### Chain of causation
1. RCCL's collective kernels carry a hidden **hostcall buffer** parameter.
2. ROCr passes that buffer only when `pcie_atomics_` is true for the device;
otherwise it returns "not supported" and the dispatch fails.
3. `pcie_atomics_` is set once, at device init, from the HSA link info between
the CPU agent and the GPU (`Device::checkAtomicSupport`). It is not re-read
per process.
4. The KFD driver populates that link info from
`pci_enable_atomic_ops_to_root()`, which — per the PCIe spec and the
kernel's implementation — requires **every upstream bridge** between the
device and its root port to advertise `AtomicOp Routing` capability
(`drivers/pci/pci.c`).
The hardware difference between the two cards:
| Card | PCIe chain (endpoint → root port) | `AtomicOpsCap` on every hop | AtomicOpsCtl |
|---|---|---|---|
| R9700 #1 (`03:00.0`) | `02:00.0` → `01:00.0` → `00:01.1` | Routing+ / Routing+ / Routing+ | `ReqEn+` (atomics ON) |
| R9700 #2 (`0b:00.0`) | `0a:00.0` → `09:00.0` → `06:09.0` → `05:00.0` → `00:02.1` | Routing+ / Routing+ / **Routing−** / **Routing−** / Routing+ | `ReqEn−` (atomics OFF) |
Card #1 is on a CPU root port; card #2 is behind the chipset bridges
`05:00.0`/`06:09.0`, which advertise `AtomicOpsCap: Routing-`. That capability
bit is **read-only** — no software can enable atomic routing through those
bridges — so the driver's atomic enablement call fails for card #2 and the KFD
marks the CPU↔GPU2 link `NO_ATOMICS`.
Writing the AtomicOp Requester bit into the endpoint's Device Control 2 by hand
(`setpci -s 0b:00.0 CAP_EXP+28.w`) does not help: the driver reads the chain at
probe time, and ROCr consumes the driver's cached answer. Only a re-probe (a
boot with the write already in place) would change it — and even then the two
`Routing-` bridges would still make `pci_enable_atomic_ops_to_root()` fail.
### What was ruled out
| Suspect | Result |
|---|---|
| Kernel version | 7.0.0-28, -30 and -31 fail identically (full `apt full-upgrade` to the newest available kernel changed nothing) |
| IOMMU | identical before and after `iommu=pt` |
| P2P hardware | fine: 45.3 GB/s bidirectional copy between the two cards; both cards pass `rocm-bandwidth-test` |
| The shipped librccl | a from-source rebuild of `ROCm/rccl` develop (with local fixes for that branch's dangling references) reproduces the same failure at the same call path — the library is not at fault |
| vLLM/pynccl | a bare two-rank `torch.distributed` allreduce with no vLLM involved fails the same way; single-rank collectives succeed on both cards (NCCL short-circuits at `world_size=1`, which is why a one-card test proves nothing) |
| Transport knobs | `NCCL_PROTO=Simple`, `P2P_DISABLE`, `SHM_DISABLE`, `CUMEM_ENABLE=0`, MSCCL off, channel counts, launch modes, `HSA_ENABLE_SDMA=0` — all identical |
| Image/support matrix | the same image on a different host with the same kernel (Proxmox VM, Codeberg issue #22) fails byte-identically |
### What would fix it
- **Move card #2 to a slot whose path is atomic-capable** (a CPU root port, as
card #1 has). This is the only software-free fix and the one that restores
true TP=2. On this AM5 platform the CPU provides the x16 slot plus x4 lanes,
so a second x16-class card necessarily lands on the chipset path — a
platform with more CPU lanes (Threadripper/EPYC) is the structural answer.
- **Done 2026-09-20 — this is exactly what happened.** The ProArt
B850-Creator WiFi NEO routes both slots to CPU root ports, the atomics
condition is satisfied, and TP=2 is the deployed shape. The Threadripper/EPYC
argument was correct about the old board and is now moot.
## Routing
The proxy decides per request, from `proxy/routing.py` (policy) and
`proxy/affinity-proxy.py` (HTTP plumbing):
- **A continuation** — a request carrying more than one message, i.e. an agent
loop re-sending its context — is pinned to the replica that served the
prefix before, because `--enable-prefix-caching` has that prefix warm there.
- **Anything else** — a one-shot question, a batch member, a burst of parallel
subagents — spreads across the cards by counter, so both prefill in parallel.
Measured on this host, 90k-token contexts and 15k-token prompts, `vllm bench`
plus a sequential-turn client:
| strategy | cold 16-request burst | warm-turn TTFT (4 turns × 2 families) |
|---|--:|--:|
| `roundrobin` (counter) | 18.6 s | 2676 ms — every other turn pays a cold prefill |
| `affinity` (hash of prefix) | 30.1 s — one card does all the prefilling | 1096 ms |
| `adaptive` (deployed) | 19.97 s | 1095 ms |
Two failures found by measuring rather than reasoning, both now covered by
`tests/frank_medium_proxy.bats`:
- Hashing a fresh prefix to pick a card sends *every* member of a same-prefix
burst to one card (29.8 s for the burst above). Only continuations may be
pinned; the spread has to come from a counter.
- The identity of a conversation is its **first 4 KiB** (`AFFINITY_BYTES`).
A prefix shorter than that window is indistinguishable from its own tail, so
turns of the same conversation hash differently and never pin.
`ROUTING=roundrobin` and `ROUTING=affinity` remain available to re-measure the
choice; the default is `adaptive`.
## Deployed configuration
Both replicas, uniform except GPU index and port:
| Knob | Value | Why |
|---|---|---|
| `TP` | 1 | one card per replica |
| `MAXLEN` | 114688 | the largest shape that boots on one card: 262144 needs 9.61 GiB of KV against 5.21 GiB available, and 131072 needs 5.58 GiB against 5.22 |
| `KV_MEM` | 0 | passed through literally as `--kv-cache-memory 0`, which the image's patched KV sizing reads as "profile this shape"; the upstream pin (2x7551-32624) is a TP=2 measurement and is invalid here. **Do not blank the variable instead of passing 0**: with the flag omitted vLLM sizes the cache from `GPU_UTIL` alone and the KV allocation dies with `torch.OutOfMemoryError` (measured: 2.16 GiB requested, 116 MiB free, 29.34 GiB already held) |
| `MAXSEQS` | 4 | matches the per-replica share of the fleet's concurrency |
| `CHUNK` | 8192 | prefill chunk (upstream default), **coupled to the context**: the prefill arena grows with the chunk size and shares the device budget with the KV cache. Measured: `CHUNK=16384` leaves 4.22 GiB of KV against the 5.21 GiB that 114688 context needs, and the engine refuses to start (`To serve at least one request with the model's max seq len (114688), 5.21 GiB KV cache is needed, which is larger than the available KV cache memory (4.22 GiB)`). Raising CHUNK therefore means cutting `MAXLEN`; single-card prefill measured ≈2.5k tok/s at 8192 (~15k-token prompt, 5992 ms TTFT) |
| `GPU_UTIL` | 0.98 | upstream default; the card is dedicated |
| `SPEC_METHOD` / `SPEC` | dflash / 7 | the z-lab DFlash2-FP8 block-diffusion drafter. Depth 7 is upstream's production default and was re-measured here on a code corpus built from this repo (see [Speculative depth](#speculative-depth)); depth 5 wins batch throughput, 7 wins single-stream latency, and this fleet is interactive |
| `spec-draft-device` | `ROCm1` | the drafter rides the card that ends up holding the target's last layers |
| `temp` / `top-p` / `top-k` | `0.7` / `0.95` / `20` | the values the vLLM predecessor served on this endpoint |
| `reasoning-format` / `reasoning-budget` | `deepseek` / `-1` | trace into `reasoning_content`, unlimited |
| `mmproj` | `mmproj-Qwen3.8-27B-Uncensored-f16.gguf` (beside the target) | vision enabled; this build has NO autoload, so the flag is explicit and the serve script refuses to start without the file |
Mounts are the TP=2 unit's minus nothing: `/models`, the HF cache, the shared
compile cache, `/patches` (the pinned radiance clone; its patches are applied
at every container start), `/r4d` (the patched libr4d — the image's shipped
`r4d.so` predates the gated-delta-net overflow fix and NaNs this model), and
the entrypoint/healthcheck/proxy scripts mounted read-only out of
`/home/dank/serverAppConfigs`.
The healthcheck reads `PORT`, so one probe serves the replicas (:7770),
the proxy (:7770) and the TP=2 target.
### Boot persistence, verified by rebooting
The trio was reboot-tested end to end (2026-09-16), because this repository has
been burned before by services that ran only from `compose.yaml` and stayed dark
after a reboot:
```
sudo reboot # unattended; ~2 min 14 s to graphical.target
# ... no manual step at any point ...
systemctl --user is-active frank-medium-a frank-medium-b frank-medium-proxy
-> active active active (containers up 59 s after boot)
# ...
curl http://127.0.0.1:7770/health # both replicas UP ~3.5 min after boot
```
Why it comes back: each unit's `[Install] WantedBy=default.target` — the generator
wires all three into `default.target`'s wants (they report `generated`, which is
the healthy state on this host's v1 quadlet generator and cannot be `enable`d) —
plus `Linger=yes` for the deploy user, so the user manager runs without a login
session. The replicas need no warm cache: the compile cache lives on disk
(`/cache`) and the model loads cold in about three minutes.
## Deploy and verify
```
./tests/run.sh # unit/config seams
./scripts/deploy.sh dankserver --service frank-medium
```
The handler re-tags `docker.io/stilldeadcode/vllm-radiance:0.9.3` as
`localhost/radiance-vllm:0.9.3` (this host's quadlet path cannot resolve
registry-prefixed refs from local storage at create time), installs the three
units, retires any stale TP=2 unit into the trash, restarts the trio and waits
for `http://127.0.0.1:7770/health`.
Verification that means something:
```
curl -s http://192.168.0.235:7770/health # proxy: names any replica that is down
curl -s http://192.168.0.235:7770/v1/models # served name must be frank-medium
# and a real completion through the proxy:
curl -s http://192.168.0.235:7770/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"frank-medium","messages":[{"role":"user","content":"Say OK"}],"max_tokens":400}'
```
The same call works against either replica directly (`:7771`, `:7772`) — when
one of them is down the other still serves, which is why the proxy reports the
individual replica states instead of a single boolean.
A replica that dies mid-flight does not cost a client a request: a failed
connection demotes that replica immediately and the proxy retries on the live
one (vLLM's endpoints are stateless, so re-sending a request that never reached
an engine is safe) and logs the failover. Verify with card 1 stopped:
```
systemctl --user stop frank-medium-a.service
for i in 1 2 3 4 5 6; do
curl -s -m 30 -o /dev/null -w "%{http_code} " -H 'Content-Type: application/json' \
-d '{"model":"frank-medium","messages":[{"role":"user","content":"probe"}],"max_tokens":8}' \
http://127.0.0.1:7770/v1/chat/completions
done; echo
podman logs --since 60s frank-medium-proxy | grep -i failover
systemctl --user start frank-medium-a.service
```
Six 200s and one `proxy: failover — … unreachable` line is the pass condition.
Restarting a stopped replica takes about 150 s (warm compile cache).
## Benchmarks
The harnesses live in the repository — the numbers below are reproducible with
`scripts/bench-vllm-corpus.py` (build the code corpus), `scripts/bench-vllm-serve.sh`
(concurrency sweep), `scripts/bench-turn-latency.py` (per-turn TTFT for an
agent-loop shape, which is what the routing decision optimizes) and
`scripts/bench-llm-soak.py` (sustained multi-family load). Each carries the
traps in its header: the tokenizer argument that makes a `vllm bench` run report
nothing, the salted prompts that keep a run from inheriting a warm cache, and
why the corpus is code-shaped rather than `random`.
`vllm bench serve` against the proxy (both cards serving), random dataset,
`--ignore-eos`, `2026-09-16`:
| Case | Output t/s | TTFT p50 | TPOT p50 | Acceptance |
|---|--:|--:|--:|--:|
| conc 1, in 1k / out 512 | 42.8 | 382 ms | 23.6 ms | 43.1 % |
| conc 4, in 1k / out 512 | 147.6 | 471 ms | 25.3 ms | 34.2 % |
| conc 8, in 1k / out 512 | 230.8 | 913 ms | 28.8 ms | 39.0 % |
| conc 1, in 32k / out 32 | 2.5 | 11837 ms | 29.6 ms | 25.0 % |
The 32k row is a prefill measurement: 32768 prompt tokens in 11.8 s is
**≈2.8k tok/s of prefill on a single card**. The replicas split the fleet's
traffic, they do not pool capacity for one request, against the upstream
reference's 4585 t/s at 32k on TP=2 — one card doing the whole prompt lands
where you would expect.
Read these against the upstream repo's own reference table (2x R9700, TP=2,
BetterBench corpus): step time 22.8-23.2 ms per update, combined decode
224 t/s, prefill 4585 t/s at 32k. Step time (TPOT) is the comparable number —
tokens/s swings with draft acceptance, and synthetic random prompts accept far
worse than the code/prose mix upstream measures (43 % here against 5.8
tokens/update there), so a lower token rate at the same step time is the
dataset, not the deployment.
Bench caveat: the client must be given `--tokenizer
/models/Qwen3.8-27B-Uncensored-MXFP4-awq` with `/models` mounted. Passing only
`--model frank-medium` makes the client try to resolve that alias on
HuggingFace and die before sending a request — a green-looking run that
produced no numbers at all.
### Speculative depth
Upstream ships depth 7 for dflash and notes the choice is content-dependent; it
was measured for TP=2 on their corpus. Re-measured here on one card against a
code corpus built from this repository's own sources (16 prompts, 512 output
tokens each, two runs per setting):
| Depth | conc 1 | conc 4 |
|---|--:|--:|
| `SPEC=7` (deployed) | **46.2 tok/s, TPOT 20.8 ms**, 3.86 tok/update | 118.7 / 127.0 tok/s, TPOT 25.1-29.3 ms |
| `SPEC=5` | 41.5 tok/s, TPOT 23.1 ms, 3.41 tok/update | **136.1 / 129.4 tok/s**, TPOT 25.3-26.6 ms |
The step *time* is nearly identical at both depths (≈79 ms against ≈80 ms at
conc 1) — depth 7 wins single-stream because it lands more tokens per step
(3.86 against 3.41), and depth 5 wins at conc 4 by ≈8 % because more, narrower
verify steps pack better. Depth 7 stays: this fleet serves about three people
in interactive agent loops, where per-turn latency is the number that is felt,
and it is also the better default on code at low concurrency. Switch a replica
to `SPEC=5` if batch throughput ever matters more than turn latency — it is one
environment line in the unit, at the cost of a recompile (the spec depth is
part of the graph cache key).
### Capacity: how many big conversations fit
Each replica's own ledger at boot states the ceiling:
```
GPU KV cache size: 122,636 tokens
Maximum concurrency for 114,688 tokens per request: 1.07x
[radiance] kv cache groups: size 8, 9 groups, 202 blocks/request (upstream would pick size 5)
Add 3 padding layers, may waste at most 60.00% KV cache memory
```
So one card holds ~122k tokens of KV — one full-context conversation plus a
little, or about two 30k-token agent loops with their cached prefixes. A 3-family
soak (three concurrent loops, each re-sending a ~30k-token document every turn
with growing history) shows what happens at that limit: turns alternate between
**~1.1 s** when the prefix is resident and **12-22 s** when it has been evicted
and must be prefilled again, because two families per card plus their cached
prefixes exceed the pool. Nothing failed — ~120 requests, no errors, no failover,
both replicas steady — and per-family pinning kept the fast turns on the replica
that holds the prefix.
Read that as a capacity budget, not a defect: two concurrent large-context
sessions per card are comfortable, three or more of that size trade turns
against each other. The soak ran 155 requests, 155 answers, no errors, no
failover and both replicas steady — the cost shows up as latency, not failure.
**`CHUNK` is not a lever for this.** The arena/KV budget is not monotonic in
the chunk size, and 8192 is a sweet spot rather than a compromise — measured
available KV per card: **5.44 GiB at `CHUNK=8192`** (122,636 tokens), 4.22 GiB
at 16384, and 4.16 GiB at 4096, where the engine refuses to start at all
(`5.0 GiB KV cache is needed, which is larger than the available KV cache
memory (4.16 GiB)`). Both directions from the deployed value cost capacity, so
the real lever for more concurrent large contexts is another card.
### Placement does not rescue an overloaded working set
If families compete for cache space, the obvious idea is to place a new
conversation on the card carrying less context instead of alternating. That was
implemented (decayed prompt-token estimate per replica, new families to the
lighter card, continuations still pinned) and measured against the counter on
the workload it should help: one ~80k-token family plus two ~50k-token
families, each turn re-sending the same document.
| placement | heavy | medium 1 | medium 2 |
|---|---|---|---|
| counter (13 min) | 13 turns, median 25.4 s, 3 cached (23 %) | 14 turns, median 34.7 s, 2 cached (14 %) | 13 turns, median 45.2 s, 2 cached (15 %) |
| balanced by load (~8 min) | 5 turns, median 46.7 s, 1 cached (20 %) | 7 turns, median 23.4 s, 0 cached | 6 turns, median 34.3 s, 0 cached |
No improvement — the load arm shows the same near-total loss of cache reuse
(note its smaller sample: the arm was stopped early once the pattern was
clear), so the change was reverted. `routing.py` keeps the counter and the
policy surface stays as small as it was.
The reason is arithmetic rather than policy: a running request's *live*
sequence and its cached prefix draw on the same pool, so a card that is running
two 50k-token generations has already exceeded its 122,636 tokens before any
cache entry is considered. Placement can decide *which* card is over, not
whether one is. Three concurrent large sessions do not fit in two cards'
capacity, and the levers are therefore scheduling (fewer large contexts in
flight, which is what the harness's context compaction does) or a third card.
### Caching itself is healthy; the sum is not
Before blaming the cache, it was measured directly: one card, an unloaded
engine, a ~50k-token prompt sent three times — **22.3 s cold, then 1.4 s, then
1.4 s** — and the same result with a varying tail between turns (22.4 s cold,
1.41 s warm). So long-prefix reuse works exactly as the pinning policy assumes,
and the engine's lifetime "Prefix cache hit rate: 29 %" line is an average over
every benchmark prompt a replica has ever seen, not a statement about any one
request.
**Serialising large requests does not fix the overload either.** Admission was
implemented — at most one >16k-token request per card at a time (`LARGE_TOKENS`
/ `LARGE_LIMIT`), small requests untouched — and run against the same
three-family workload:
| | heavy | medium 1 | medium 2 |
|---|---|---|---|
| no admission | 13 turns, median 25.4 s, 23 % cached | 34.7 s, 14 % | 45.2 s, 15 % |
| one large per card | 5 turns, median 48.2 s, 20 % cached | 22.7 s, 14 % | 22.6 s, 33 % |
No clear win on small samples, so it was reverted too. The reason is now
visible in the arithmetic: with three families on two cards, one card must hold
**two** families, and its pool has to carry 2 × 50k of cached prefix *plus* the
50k of the sequence currently generating — 150k against 122,636. Those two
co-tenants evict each other turn by turn whatever the concurrency limit is. A
card can hold two large families' caches, or one cache plus a live generation,
but not both. That is the wall, and it is capacity, not scheduling.
### Watching it instead of measuring it by hand
Everything above was diagnosed with ad-hoc probes. The same signals are now
scraped — Prometheus has `frank-medium` and `frank-dragoon` jobs
(`shared/prometheus/prometheus.yml`), so the numbers that matter have history:
`vllm:kv_cache_usage_perc` (per-card pool fill), `vllm:num_requests_waiting`
(queue depth), `rate(vllm:prefix_cache_hits_total[5m]) /
rate(vllm:prefix_cache_queries_total[5m])` (whether pinned conversations are
served warm), and the request-latency histograms. See [[Observability]], and one
Grafana dashboard reads them
(**LLM fleet — capacity and reuse**).
Five rules watch this service (`shared/prometheus/rules/llm-fleet.yml`): pool
saturation above 90 %, requests queueing for 10 minutes, prefix reuse collapsing
below 20 % with traffic, the scrape target down, and p95 latency above 60 s —
each threshold tied to a measurement on this page rather than to a guess. They
evaluate and show up on the dashboard's **Alerts firing** panel, but **nothing
is notified**: this stack has no Alertmanager, so an alert reaches a person only
if that person is looking. That is the honest state of "alerting" here.
## Citations
- vLLM upstream: https://github.com/vllm-project/vllm — the R4D one-shot
allreduce and the tensor-parallel path this deployment runs.
- `ROCm/clr` `rocclr/device/rocm/rocvirtual.cpp:3963` — the hostcall-buffer
refusal; `rocclr/device/rocm/rocdevice.cpp:186` — `checkAtomicSupport`.
- Linux `drivers/pci/pci.c` — `pci_enable_atomic_ops_to_root()` and its
`PCI_EXP_DEVCAP2_ATOMIC_ROUTE` requirements on upstream bridges.
- Codeberg `StillDeadcode/vllm-radiance` issue #22 — an independent host with
the same image and kernel 7.0.0-30 failing identically.
- `ggz14/radiance-vllm-mxfp4` — the patch set and the reference benchmark table
this page compares against.
serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md
-220
View File
@@ -1,220 +0,0 @@
---
type: configuration
title: frank-medium ROCmFPX Runbook
description: charlie12345/ROCmFPX llama.cpp serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF as frank-medium on Dankserver (2x AMD Radeon AI PRO R9700, gfx1201).
tags:
- frank-medium
- dankserver
- rocmfpx
- llama-cpp
- mtp
- gfx1201
- r9700
timestamp: 2026-09-15
---
> **PRIOR deployment (retired 2026-09-16).** This page documents the
> llama.cpp era of `frank-medium`, which served the endpoint for part of
> 2026-09-15 before the vllm-radiance MXFP4 stack replaced it. The ACTIVE
> deployment is [[frank-medium-llamacpp-dflash2]]; this page is kept as the
> record of the runtime it replaced.
# frank-medium ROCmFPX Runbook
`frank-medium` is the Dankserver LLM endpoint (port 7770) since 2026-09-15,
when the RTX 3090 left the box and two **AMD Radeon AI PRO R9700** cards
(Navi 48, RDNA4, `gfx1201`, 32,624 MiB each, PCIe 5.0 x16) took its place.
It replaced the retired FreeToken/CUDA `frank-small`
([[frank-zealot]]), which cannot run on this hardware at all.
Stack: **charlie12345/ROCmFPX llama.cpp** (`main`, `c49ebdb`, 2026-08-22)
serving **kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF** with the
companion MTP draft head. Stock llama.cpp cannot load this GGUF —
`Q8_0_ROCMFPX_AGENT` is ftype 115 / `GGML_TYPE_Q8_0_ROCMFPX` (type 103) and
exists only in the fork.
## Hardware
| Property | Value |
|----------|-------|
| GPUs | 2x AMD Radeon AI PRO R9700 (DID 0x7551), rev c0 |
| Architecture | gfx1201 (Navi 48, RDNA4) |
| VRAM | 32,624 MiB each (rocm-smi) |
| Link | PCIe, no XGMI; cards sit in different IOMMU groups |
| iGPU | Raphael (gfx1036) enumerates as ROCm device 2 — hidden from HIP |
| CPU / RAM | Ryzen 7 7700X (8c/8t), 60 GiB RAM, 63 GiB swap |
| Kernel driver | in-tree `amdgpu`; no new driver needed |
`HIP_VISIBLE_DEVICES=0,1` hides the iGPU so nothing can drift onto shared
system RAM. Devices are addressed as `ROCm0,ROCm1` (llama-server) or
`ROCm0/ROCm1` (llama-bench multi-device syntax — see the bench gotcha below).
## Runtime build (self-contained ROCm 7.14)
The host ROCm install is quarantined rather than repaired: the build uses the
fork's self-contained toolchain, so the binary carries its own
`libamdhip64`/`libhsa-runtime64`/`rocblas`/`hipblaslt` with
`RPATH=$ORIGIN/../lib` and never touches system ROCm.
```bash
cd /home/dank/GitPackages/charlie-ROCmFPX
env JOBS=8 ./scripts/build-rocmfp4-rocm714-local.sh
# -> build-rdna4-rocm714/bin/llama-server
```
The script downloads the gfx120X-all TheRock ROCm 7.14 nightly tarball into
`~/.local/share/rocm-7.14.0a20260624` (~17 GiB extracted; tarball kept next to
the build tree) and builds `gfx1200;gfx1201`.
**gfx1201 is not interchangeable with gfx1200** — the fork's
`docs/BUILD-AMD-ARCHITECTURES.md` warns a wrong-sibling build links, loads the
model, then segfaults. `scripts/rocmfpx-hip-arch.sh` auto-detects and refuses.
## Model
| File | Size | Role |
|------|------|------|
| `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` | 28,214,425,952 B (26.28 GiB, 8.39 bpw) | target |
| `mtp-Qwen3.8-27B-Q4_0.gguf` | 1,680,271,648 B | MTP draft head |
| `mmproj-Qwen3.8-27B-BF16.gguf` | 931,145,888 B | vision tower — **unused** (`--no-mmproj`) |
The AGENT routing keeps 13 of the 17 `*.output.weight` tensors at full Q8_0
(12 more than the plain `Q8_0_ROCMFPX` build), which raises MTP draft
acceptance (0.953 vs 0.911 on the author's gfx1151 box). Take this variant
only when serving with MTP. Fetch with `scripts/download-q8agent.sh`
(resumable, size-verified; run `--verify-only` to re-check).
The model card documents 131,072 context; llama.cpp reports
`n_ctx_train (262144)` from the GGUF, but the author never tested beyond 131k,
so the deployment uses 131,072.
## Parallelism: layer split, not tensor (measured)
llama-bench on the host (pp4096 / tg128, same at depth 32768, f16 KV,
flash-attn on, 3 reps):
| Split | pp4096 | tg128 | pp4096 @d32768 | tg128 @d32768 |
|-------|-------:|------:|---------------:|--------------:|
| 1x R9700 (`-sm none`) | 1160.9 | 19.42 | ~512 | ~18.4 |
| **2x layer (`-sm layer`)** | **1991.2** | 18.76 | **859.1** | 17.64 |
| 2x tensor (`-sm tensor`) | 271.8 | 26.26 | 228.6 | 25.09 |
Tensor parallel wins decode by ~40 % but collapses prefill **7x**: this build
ships without RCCL, so llama.cpp logs
`internal AllReduce init failed (n_devices != 2?); falling back to meta-backend
butterfly` and every layer pays a slow cross-card reduction. The workload is
prefill-heavy (agentic, long prompts, reasoning traces), so the operator
switched to layer split on 2026-09-15. Revisit if the fork ships a working
RCCL tensor path (`-DGGML_HIP_RCCL=ON`).
Served-path comparison with MTP on (real completions): layer 35.7–46.2 tok/s
vs tensor 28.7 tok/s — layer wins decode too once the draft head is in play.
### llama-bench multi-device gotcha
`llama-bench` and `llama-server` disagree on device syntax. In llama-bench,
`-dev ROCm0,ROCm1` means *two separate single-card benches* (the value is
split on `,` into separate device sets, one model load each); a single split
bench needs `-dev ROCm0/ROCm1`. The invalid form silently produces
single-card numbers in a table that looks like a split test.
### Tensor-mode draft abort (if tensor is ever revisited)
Under `--split-mode tensor` the MTP draft head must be pinned to exactly one
device (`--spec-draft-device ROCm0`); with it empty the draft joins the
meta-device KV allocation and the server aborts while building the KV cache:
```
ggml/src/ggml-backend.cpp:119: GGML_ASSERT(buffer) failed
<- ggml_backend_meta_alloc_ctx_tensors_from_buft <- llama_kv_cache ctor
```
## Deployment
| File | Role |
|------|------|
| `scripts/frank-medium.env` | all tunables (sampler, split, draft, context) |
| `scripts/serve-frank-medium.sh` | env-driven llama-server launcher |
| `scripts/frank-medium.service` | root systemd unit (relinked into /etc/systemd/system) |
| `scripts/download-q8agent.sh` | model fetch + size verification |
```bash
./scripts/deploy.sh dankserver --service frank-medium
```
First deploy (2026-09-15) also ran `retire_root_llm_small()` in
`scripts/deploy-remote.sh`: the CUDA-era `frank-small` unit was stopped,
disabled and moved to `~/.AGENTTRASH/20260915-frank-zealot-retire/host-units`
(the host's `systemctl disable` has been observed to delete unit files,
2026-08-26 incident, so both outcomes are tolerated).
Deployed flags (see `scripts/frank-medium.env` for the full set):
`--split-mode layer --device ROCm0,ROCm1 --gpu-layers 999 --fit off`,
`--ctx-size 131072 --parallel 1`, f16 K/V, flash-attn on,
`--spec-type draft-mtp` (n_max 6, p_min 0.6) with the draft pinned to
`ROCm1` — the card holding the target model's last layers, where the final
hidden state is produced. Reasoning on, deepseek format, unlimited budget;
sampler defaults are the checkpoint's instruct table (temp 0.7, top-p 0.8,
top-k 20, min-p 0, presence 1.5). Text only — vision stays on
`frank-dragoon` (Dankclaw).
## Verified serving behavior (2026-09-15)
- Health `{"status":"ok"}` ~18 s after start; model load ~17 s (warm page cache).
- MTP engaged: draft acceptance 60–69 % across prompts
(`timings.draft_n` / `draft_n_accepted` in every chat response).
- Decode 35.7–46.2 tok/s on a 200-token budget; cold prompt 949 tokens
prefilled in 2036 ms (~466 tok/s at ubatch 512).
- **Prefix caching works with the draft head**: the model card's
`mtp-prompt-cache-fix.patch` defect (draft head disables prefix caching) is
already fixed in `c49ebdb` — the speculative boundary rides inside the
context checkpoint (`common_prompt_checkpoint::data_spec`). Measured: turn 2
of a conversation reprocessed 22 tokens (`cache_n` 945) instead of 949.
- Quirk: the request-level `"speculative": {"n_max": 0}` override does **not**
disable drafting (still drafts and accepts). To serve without MTP, restart
with `SPEC_TYPE=none`.
## Client contract
OpenAI-compatible `/v1` on port 7770, alias `frank-medium`. Reasoning traces
are projected into `reasoning_content` (deepseek format); requests that send
no sampler fields get the server's CLI defaults, not greedy. Text only.
Harness routing after the cutover: blankpad `smol`/`tiny`/`commit`/`default`
→ `frank-medium`, Hermes `title_generation` (and its other dankserver-small
routes) → `frank-medium`, vision stays `frank-dragoon`.
## Host ROCm posture (post-cleanup)
Historically the host had a broken mix: AMD-repo ROCm 6.3.3 in
`/opt/rocm-6.3.3` hijacked `libhsa-runtime64` via
`/etc/ld.so.conf.d/10-rocm-opencl.conf`, breaking Ubuntu's ROCm 7.1 userspace
(`undefined reference to 'hsa_amd_memory_get_preferred_copy_engine'`).
On 2026-09-15 the stale 6.3.3 packages (~24.04-suffixed, 167 pkgs) were
purged, the hijack conf and the `repo.radeon.com` apt pin moved to
`~/.AGENTTRASH/20260915-rocm633-cleanup/`, and `ldconfig` now resolves only
Ubuntu's ROCm 7.1 libs. `rocm-smi` is Ubuntu's own 7.1.1 package. The
self-contained toolchain in `~/.local/share/rocm-7.14.0a20260624` does not
depend on any of it.
## Troubleshooting
- **Wrong arch build**: a gfx1200 binary on this card links, loads, then
segfaults. Rebuild with the pinned script and check the "Build complete for
gfx1200;gfx1201" line plus the emitted code objects.
- **Crash at load**: read the journal (`sudo journalctl -u frank-medium -n 100`),
then run the serve script directly; the bundled libs make the script and the
raw binary behave identically. The known abort shape is the tensor-mode
draft KV abort above.
- **MTP "not engaged"**: check `timings.draft_n` in a response; zero means the
draft head is not loaded (look for `common_speculative_init: adding
speculative implementation 'draft-mtp'` in the journal).
- **Model files missing/truncated**: `bash scripts/download-q8agent.sh
--verify-only`.
## See Also
- [[frank-zealot]] — the retired predecessor (FreeToken/CUDA on the 3090)
- [[kingjones-q8agent-rocmfpx]] — the gfx1151 investigation of this checkpoint family
- [[Llama Cpp Runtimes]] — where the ROCmFPX fork sits among the runtimes
- [[Dankserver System Specs]] — current hardware
- [[deployment]] — the deploy flow
serverAppConfigs-docs/LLM/frank-npu.md
-164
View File
@@ -1,164 +0,0 @@
---
type: reference
title: frank-npu (retired) — FLM NPU endpoint on Dankclaw
description: The retired FastFlowLM endpoint on Dankclaw's XDNA2 NPU — what it served, why it died on 2026-09-11 with the amd_iommu=off decision, and where every dependent route went.
tags:
- flm
- fastflowlm
- xdna2
- npu
- dankclaw
- frank-npu
- retired
timestamp: 2026-09-11
---
# frank-npu — the retired FLM NPU endpoint
> **RETIRED 2026-09-11 by operator decision** — and still retired: there is no
> NPU endpoint service on Dankclaw today. The IOMMU configuration it required
> has since been reversed twice — see
> [[dankclaw-iommu-translation-off]]: the live 2026-10-18 posture is
> `cat /proc/cmdline` → `… ro quiet splash iommu=pt …` (no `amd_iommu` token),
> `amdxdna` is bound and `/dev/accel/accel0` exists, so the hardware is
> reachable again — revival is tracked as Gitea issue #3. The service unit, env
> file and serve script are staged in `serverAppConfigs/.AGENTTRASH/`
> (suffixed `.flm-npu`), and the host unit is still masked
> (`systemctl is-enabled frank-npu` → `masked`, read 2026-10-18). Port 7771
> belongs to [[frank-zealot]] today.
The third inference endpoint on Dankclaw: a FastFlowLM (FLM) v1.0.1 server
(`/usr/bin/flm`, dpkg `fastflowlm`) on the **XDNA2 NPU** (50 TOPS), serving the
whole local model registry on port 7771 and routing per request by the exact
FLM registry tag. Its one capability the iGPU fleet lacked was **sight** — and
briefly **sound**: the Gemma4 E4B IT NPU2 pack carries both a vision and an
audio tower, which is why the service ran that pack rather than the
uncensored-but-text-only alternative.
This page is kept as the retirement record. The operational detail below was
live-verified while the service ran and remains accurate about FLM's shape.
## Why it is retired
The replacement large-model runtime is worth **13–16 % of prefill** if the
kernel boots with `amd_iommu=off` instead of `iommu=pt`, measured by the
upstream project on this exact silicon. That single flag is the whole decision:
- With IOMMU off there is no PASID/SVA state, and the XDNA2 driver cannot hand
out `/dev/accel/accel0` — every open fails with `err=-19` ("No such device").
That is precisely the failure the endpoint was rescued from on 2026-08-30,
recorded in the next section.
- `iommu=pt` was recorded here as "not a middle ground", on the theory that the
passthrough domain still costs the vendor's prefill number. **That claim is
dead: the host flipped to `iommu=pt` on 2026-10-18 and the local A/B on
byte-identical traffic measured the cost at nothing** (+2.0 % cold prefill,
1,616 t/s at 59k tokens). Identity mode does keep a 1:1 page walk plus
ATS/PRI/PASID/GCR3 armed for the iGPU — worth 2–4 % on this silicon class, and
**AMD PerfOpt** (Linux 7.4, merged 2026-09-24) removes it outright by enforcing
only the IR/IW permission bits for the privileged iGPU — but nothing waits on
that now.
- **The precondition is satisfied; the endpoint is still not deployed.** `amdxdna`
binds `0000:c6:00.1` (driver 0.7.0, firmware `amdnpu/17f0_11/npu_7.sbin`), the
NPU sits in IOMMU group 26 `identity` with PASID width 10, `/dev/accel/accel0`
is `crw-rw---- root:render 261,0` and `dank` is in `render` — so a revival
quadlet needs `AddDevice=/dev/accel/accel0` + `GroupAdd=keep-groups`, the
ComfyUI `/dev/kfd` pattern. Re-read live 2026-10-18: `lspci -k -s c6:00.1` →
`Kernel driver in use: amdxdna`; `journalctl -k -b` → `Initialized
amdxdna_accel_driver 0.7.0 for 0000:c6:00.1` and `Load firmware
amdnpu/17f0_11/npu_7.sbin`; `/sys/kernel/iommu_groups/26/type` → `identity`;
`ls -l /dev/accel/accel0` → `crw-rw---- 1 root render 261, 0` (created
2026-10-18 01:25); `id -nG dank` contains `render`. Kernel versions to avoid:
**6.18–6.18.7** disables SVA outright *(unverified 2026-10-18 — upstream
reports place the amdxdna/Strix-Halo breakage at 6.18.4+ and fixed by 6.19.2,
not at a clean 6.18–6.18.7 window)*. Full census in
[[dankclaw-iommu-translation-off]] § *2026-10-18 execution*.
- The operator took the vendor's prefill figure on the September evidence, so the
NPU endpoint went — and it stays retired because nobody has asked for a model
on it, not because the hardware is unreachable.
Consequences accepted at retirement: **audio input left the fleet with no
replacement** (halogen reads images only, no audio or video), and the surviving
image path is halogen's vision tower on `frank-dragoon`.
## Where its routes went
| consumer | setting | now |
|---|---|---|
| Hermes | `auxiliary.vision` | `dankclaw-large` / `frank-dragoon` (halogen vision tower) *(unverified 2026-10-18: `/opt/hermes` is gone from Dankclaw, so the live `config.yaml` could not be read)* |
| Hermes | `auxiliary.title_generation` | `dankserver-small` / `frank-zealot` *(unverified 2026-10-18: same)* |
| harness (`~/.omp/agent/config.yml`) | `modelRoles.vision` | moved again: `llamacpp (dankclaw-3090)/frank-zealot:xhigh` (`config.yml:6`) |
| harness | `modelRoles.tiny` | `llamacpp (dankclaw-3090)/frank-zealot:off` (`config.yml:10`) |
| harness | `modelRoles.task` | `llamacpp (dankclaw-3090)/frank-zealot:off` (`config.yml:11`) |
The `flm (dankclaw-npu)` provider block was removed from the harness model
config, and the `dankclaw-npu` provider from Hermes's live `config.yaml`.
## What it was
| Field | Value |
|---|---|
| Runtime | FLM v1.0.1 (`/usr/bin/flm`, dpkg `fastflowlm`; `dpkg -l fastflowlm` on 2026-10-18 still reports `fastflowlm 1.0.1`) on the XDNA2 NPU |
| Service | `frank-npu.service` (systemd **system** unit, `User=dank`, `Restart=on-failure`, `RestartSec=10`, `LimitMEMLOCK=infinity`) |
| Files (archived) | `scripts/frank-npu.env`, `scripts/serve-frank-npu.sh`, `scripts/frank-npu.service` |
| Port | `7771` on `0.0.0.0` (that port is frank-zealot's today) |
| Power mode | `FLM_PMODE=turbo` — nothing else used the NPU, so the top power mode cost no neighbour anything |
| Context | `FLM_CTX_LEN=131072` (128K, the verified maximum on this box) |
| Vision/audio | `FLM_ASR=0`; the E4B pack answers images through its own vision tower, and `FLM_ASR=1` would have added Whisper-V3-Turbo-NPU2 (+0.62 GB) and `/v1/audio/transcriptions` |
The serve script exported `FLM_CONFIG_PATH=/home/dank/.config/flm/model_list.json`
and `FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins`, with kernels symlinked
from `/opt/fastflowlm/share/flm/xclbins/` into the per-pack directories, so a
tag swap was an env-file edit rather than a script edit.
## API contract (as it behaved)
- The endpoint served the **whole local registry** (36 tags at first deploy) on
one port; `/v1/models` listed them all. Chat completions routed per request
by the exact registry tag in the `model` field — never a service alias.
- **Unknown tags did not 404**: `/v1/chat/completions` answered HTTP 200 with a
flat error body `{"error":"Failed to load <tag> model!"}` and no `choices`.
Status-code-only clients silently got an empty response.
- Health was `GET /v1/version`; FLM has **no `/health` route**.
- The NPU is a single-lane device — generations serialized FIFO. It was a
background/triage endpoint, not interactive multi-user chat.
## The IOMMU/SVA root cause (first deploy crashed, fixed 2026-08-30)
First deploy crash-looped every 10 s with
`Error: Open /dev/accel/accel0 failed (err=-19): No such device`. The chain,
all live-verified on the host:
1. `amdxdna_drm_open` failed **SVA bind with ret -19** for every client — the
XDNA2 driver cannot hand out the accel device without PASID/SVA.
2. SVA requires the AMD IOMMU to be enabled, and the kernel cmdline then
carried `iommu=off`. With IOMMU off there are no IOMMU groups and no PASID
state; `PASIDCtl: Enable-` was observed on `0000:c5:00.1`.
3. The fix was `iommu=pt`: passthrough keeps the iGPU default domain
identity-mapped (no DMA-translation overhead for `frank-dragoon`) while
enabling the IOMMU hardware and PASID that the driver's SVA bind needs.
4. Post-boot verification: `/sys/kernel/iommu_groups/` populated (30 groups),
zero `SVA bind device failed` entries since boot, `accel0` recreated at
boot, service active with `NRestarts=0`.
The same root cause explained the older
`amdgpu 0000:c4:00.0: no vmid pasid mapping supported` kernel line — a
consequence of IOMMU-off, not a GPU fault.
**The standing rule this produced — "never set `iommu=off` on this host again" —
was superseded twice**: `amd_iommu=off` was the intentional posture from
2026-10-16 to 2026-10-18, and the host has since flipped back to `iommu=pt`
(live `cat /proc/cmdline`, 2026-10-18: `… ro quiet splash iommu=pt …`). See
[[dankclaw-iommu-translation-off]] for the current posture and how to revert it.
## Client wiring (historical)
The pi/harness model config registered this endpoint with the **exact** FLM
registry tag as the model `id` and `baseUrl` `http://192.168.0.143:7771/v1`.
Both that provider block and Hermes's `dankclaw-npu` provider were removed at
retirement; the routes are mapped in the table above.
## Citations
- [FastFlowLM](https://github.com/FastFlowLM/FastFlowLM) — FLM runtime and NPU model packs
- [[dankclaw-iommu-translation-off]] — the decision that retired this endpoint
- [[frank-dragoon]] — the surviving image-input path
serverAppConfigs-docs/LLM/frank-redo-2026-08-17.md
-225
View File
@@ -1,225 +0,0 @@
---
type: reference
description: "This page records the full redo of the inference fleet on 2026-08-17."
tags:
- llm
- frank
timestamp: 2026-09-03
---
# Frank Program Redo — 2026-08-17
This page records the full redo of the inference fleet on 2026-08-17.
The pipeline-parallel split and the FastFlowLM NPU endpoint were retired.
The old scripts moved to `old/scripts/` in this repository.
> **HISTORY — the topology below is two fleet generations dead.** Today the
> fleet is three endpoints, each owned by its page: [[frank-archon]]
> (`192.168.0.235:7770`), [[frank-dragoon]] (`192.168.0.143:7770`) and
> [[frank-zealot]] (`192.168.0.143:7771`). No vLLM and no ROCm endpoint serves
> anything; there is no pipeline parallelism and no NPU endpoint.
## New Topology
| Service | Host | Runtime | Model | Port |
|---------|------|---------|-------|------|
| frank-dragoon | Dankserver | llama.cpp (patched FastMTP) | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS | 7770 |
| frank-zealot | Dankclaw | vLLM ROCm | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8 | 52625 |
Names as the 2026-08-17 commit wrote them: `frank-default-large` on Dankserver
(`git show d9af7f1:scripts/frank-default-large.env:20,22` → `ALIAS=frank-default-large`,
`LISTEN_PORT=7770`) and `frank-default-small` on Dankclaw
(`git show d9af7f1:scripts/frank-default-small.env:16,18` → `ALIAS=frank-default-small`,
`LISTEN_PORT=52625`); the Protoss ids date from 2026-10-06 → 10-10.
The RTX 3090 runs the whole 27B model standalone. No RPC worker, no
tensor split, no NPU.
## Models
### frank-dragoon — HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF:IQ4_XS
- Base: Qwen3.8-27B (dense 27B, 64 layers: 48 GatedDeltaNet + 16
gated-attention, hidden 5120, 24 Q heads / 4 KV heads, head dim 256,
native 262144 ctx).
- Quant: IQ4_XS (4.60 BPW, 15.71 GB). Every GGUF in the repo preserves the
native embedded MTP/NextN head.
- FastMTP draft sidecar: `...FastMTP-32K.gguf` (903 MB). Requires the
patched llama.cpp build (commit `4df29be4f4c3673f428170fda944a5b19f743bb8`
+ `HauhauCS-FastMTP-llama.cpp.patch`). Without the patch, the draft load
fails with `expected 5120, 248320, got 5120, 32768`.
- Spec chain: FastMTP (`--spec-type draft-mtp`, `--spec-draft-n-max 2`,
`--spec-draft-ngl all`, `--spec-draft-p-min 0`) then ngram-mod fallback
(48/64/24).
### frank-zealot — nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8
- Base: Qwen3.6-35B-A3B (hybrid MoE, 256 experts top-8, ~3B active, 40
layers: 10 blocks of 3 DeltaNet + 1 Attention, native 262144 ctx).
- Quant: AMD Quark W8A8 INT8 (per-channel static weights, per-token
dynamic activations). MoE experts stored as per-expert nn.Linear
triplets that match vLLM's FusedMoE mapping exactly. Weights ~35 GB;
the vision tower, router, lm_head stay BF16.
- Served with vLLM ROCm, ngram speculative decoding, and tool calling.
- Swap history: the initial deployment used
`Avesed/Qwen3.6-35B-A3B-INT8-W8A8` (compressed-tensors INT8). Its tool
calls were malformed (the model emitted `<function=get_weather(city=...)>`
instead of the Qwen3 XML structure). The Quark quant follows the format
cleanly; swapped in the evening of 2026-08-17.
## Context Math (frank-dragoon, RTX 3090)
KV per token: 16 attention layers x 4 KV heads x 256 dim.
K q8_0 = 1088 B per layer (4 KV heads × 256 dim × 1.0625 B), V f16 = 2048 B per
layer, ×16 layers = 50176 B/token total.
Budget (MiB): 24576 total - 14982 weights - 861 draft - 1024 fixed
(CUDA context, compute graph, scratch, draft KV) - headroom.
| Context | KV (MiB) | Free after load |
|---------|----------|-----------------|
| 114688 (112K) | 5488 | 2221 (2.17 GiB) |
| 122880 (120K) | 5880 | 1829 (1.79 GiB) |
| 131072 (128K) | 6272 | 1437 (1.40 GiB) |
| 147456 (144K) | 7056 | 653 (0.64 GiB) |
Production choice: **131072 (128K)** — the maximum that keeps about
1.4 GiB headroom for checkpoints. Lower `CTX_SIZE` in
`scripts/frank-dragoon.env` to 122880 for ~1.8 GiB headroom.
## Install Runbook
### Dankserver (frank-dragoon)
1. Stop the old worker: `sudo systemctl stop frank-pp-worker`.
2. Download the model and draft to `/home/dank/models/hauhaucs-qwen38/`.
3. Build the patched llama.cpp:
```
git clone https://github.com/ggerganov/llama.cpp /home/dank/llama.cpp-fastmtp
cd /home/dank/llama.cpp-fastmtp
git checkout 4df29be4f4c3673f428170fda944a5b19f743bb8
curl -L -o HauhauCS-FastMTP-llama.cpp.patch <HF patch URL>
git apply HauhauCS-FastMTP-llama.cpp.patch
cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release \
-DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON
cmake --build build --config Release -j$(nproc)
```
`GGML_CUDA_FA_ALL_QUANTS=ON` is mandatory. Without it, prefill collapses
~12x (the q8_0 KV + flash attention falls back to a slow path;
measured 95 t/s vs 1300+ t/s at 8K on the 3090).
4. Symlink the unit, reload, enable, restart:
```
sudo ln -sf /home/dank/vLLM/scripts/frank-dragoon.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable frank-dragoon
sudo systemctl restart frank-dragoon
```
5. Verify: `curl -sf http://127.0.0.1:7770/health` and a chat completion.
6. Confirm VRAM headroom with `nvidia-smi`.
### Dankclaw (frank-zealot)
1. Stop the old services: `sudo systemctl stop frank-dragoon
frank-zealot` (retired PP head and NPU endpoint).
2. Install uv and create the venv:
```
curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv --python /usr/bin/python3.12 /home/dank/vllm-venv
uv pip install --python /home/dank/vllm-venv vllm --torch-backend=auto
```
The resolver pulls the AMD ROCm wheel set (torch +
amd-torch-device-gfx1151 + the vllm rocm variant) from the AMD index.
3. Cache the model (default HF cache):
```
python3 -c "from huggingface_hub import snapshot_download; snapshot_download('Avesed/Qwen3.6-35B-A3B-INT8-W8A8')"
```
4. Install the ROCm wheel set and the flash-attention triton path:
```
uv pip install torch --torch-backend=auto
uv pip install "vllm==0.27.1+rocm723" --index-url https://wheels.vllm.ai/rocm/ \
--extra-index-url https://pypi.org/simple --index-strategy unsafe-best-match --no-build-isolation
cp -r <ciru-venv>/site-packages/flash_attn* <vllm-venv>/site-packages/
# add a flash_attn_2_cuda stub module (ROCm build has no compiled .so)
```
Set `FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE` (env file). Without it the
vision encoder falls back to torch SDPA and engine init OOMs.
5. Symlink the unit, reload, enable, restart:
```
sudo ln -sf /home/dank/vllmScripts/scripts/frank-zealot.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable frank-zealot
sudo systemctl restart frank-zealot
```
6. Verify: `curl -sf http://127.0.0.1:52625/health` and a chat completion.
## Fallback: vLLM source build for gfx1151
If the prebuilt ROCm vLLM wheel cannot run on gfx1151, build from source.
The retired CIRU package documents the proven recipe:
`/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/scripts/build-vllm-gfx1151.sh`
(branch `ciru/ling3-gfx1151-rocm715`, index `https://rocm.nightlies.amd.com/whl-multi-arch/`).
The upstream starting point needed a five-line ROCm sampler fix to launch
on gfx1151; newer vLLM releases may already include it.
## Configuration
- `models.json` (pi): two providers — `llamacpp (dankserver)` at
http://192.168.0.235:7770/v1 and `vllm (dankclaw)` at
http://192.168.0.143:52625/v1. (The harness config today has three providers
and no vLLM: `~/.omp/agent/models.yml:14,94,173`.)
- HF token: `~/.cache/huggingface/token` on both machines (mode 600).
- OpenWebUI (sibling repo): retarget `OPENAI_API_BASE_URL` to the
Dankserver endpoint.
## Evening Session (2026-08-17): tool calling, model swap, batch 8192
Three changes landed after the initial deployment:
### 1. Tool calling on frank-zealot (HTTP 400 fix)
Clients (pi) send `tool_choice: "auto"`. vLLM rejects it with HTTP 400
unless the server enables tool calling. Fixes applied:
- `--enable-auto-tool-choice --tool-call-parser qwen3_xml` (the Avesed /
Quark chat template renders the Qwen3 XML tool format, and qwen3_xml
is the matching parser).
- Custom chat template `/home/dank/models/qwen36-chat_template-tools.jinja`:
derived from the repo's `chat_template.jinja` (the tokenizer_config.json
ships no template, so vLLM would use its built-in one, which does not
render the XML tool instructions). The custom template adds:
- a concrete WORKED EXAMPLE built from the actual tool schema
(name + parameter keys) — models follow concrete examples much
better than the generic placeholder;
- tool requests default to non-thinking (`enable_thinking` unset + tools
present -> empty think block). Qwen3 tool calls only work reliably
with thinking off; normal requests keep thinking.
Known pitfall: the earlier Avesed model still emitted malformed XML even
with the strong template (`<function=get_weather(city=...)>`); that was
model behavior, which motivated the swap.
### 2. Model swap: Avesed INT8 W8A8 -> Quark W8A8 INT8
`frank-zealot` now serves
`nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8` (AMD Quark 0.11.1, ~35 GB,
vLLM-native expert layout). The Avesed compressed-tensors quant's
Qwen3-XML tool fidelity was poor. Verified with the Quark model: plain
tool requests (no chat_template_kwargs) return clean
`tool_calls: [{"name": "get_weather", "arguments": {"city": "Paris"}}]`;
non-tool requests keep reasoning. The Avesed checkpoint stays in the HF
cache for rollback.
### 3. Batch 8192
`MAX_NUM_BATCHED_TOKENS` lowered from 16384 to 8192 (user request).
vLLM has no separate ubatch knob; the scheduler issues one batch per
step up to this token cap.
### 4. Prefix caching enabled
Added `--enable-prefix-caching`. The model is hybrid (GatedDeltaNet +
full attention); vLLM keeps prefix caching **opt-in** for hybrid models
(`default_prefix_caching = is_prefix_caching_supported and not
is_hybrid`), so it was off by default. Enabling it reuses the KV of
repeated prompt prefixes. Measured on a 2023-token prefix (2026-08-17
evening): 4.6s cold -> 1.1s cached (~4x TTFT), output unchanged. This is
the main TTFT lever for pi, which re-sends the same system prompt + tool
schema every turn. Toggled via `ENABLE_PREFIX_CACHING` in the env file.
serverAppConfigs-docs/LLM/frank-zealot.md
+29 -27
View File
@@ -1,7 +1,7 @@
---
type: reference
title: "frank-zealot on Dankclaw — llama.cpp (PrismML fork 10777), Ternary-Bonsai-2-9B PQ2_0-MTP, 3 × 200704, f16 K + q8_0 V, MTP armed"
description: "The one frank-zealot reference: the live shape on Dankclaw's RTX 3090 (:7771) exactly as scripts/frank-zealot.env ships it, every era with its rollback path, the deduplicated laws of this card, the wire contract, and the owning files."
description: "The RTX 3090 endpoint: live shape exactly as the env ships it, every era with its rollback path, and its card laws."
tags:
- llm
- frank-zealot
@@ -9,13 +9,14 @@ tags:
- llama-cpp
- gguf
- rtx3090
- endpoint
timestamp: 2026-10-18
---
# frank-zealot — llama.cpp (PrismML) Bonsai-2-9B PQ2_0-MTP, 3 × 200,704, f16 K + q8_0 V, MTP armed
Dankclaw's RTX 3090 endpoint — `http://192.168.0.143:7771`, wire id `frank-zealot`, the fleet's **compaction hub**
(three conversations at 200,704 each, the window the other two endpoints also cap at). One page per
(three conversations at 200,704 each, the window the other two endpoints also cap at). One page
endpoint: eras are rows below, not pages. Siblings: [[frank-archon]],
[[frank-dragoon]]. Deploy with `./scripts/deploy.sh dankclaw --service frank-zealot`; swap mechanics in
[[Model Swap and Rollback]], sizing math in [[Llama Cpp Serving Sizing]]. `scripts/frank-zealot.env` and
@@ -34,7 +35,7 @@ is the bug.
| Address / alias | `BIND_ADDR=0.0.0.0`, `LISTEN_PORT=7771`, `ALIAS=frank-zealot` (a flag, not a fact) |
| Flags | `--ctx-size 602112 --parallel 3` ⇒ 200,704 per slot · `--cache-type-k f16 --cache-type-v q8_0` (asymmetric, **14,406.00 MiB**) · `--flash-attn on` · `--batch-size 6144 --ubatch-size 1024` · `--n-gpu-layers 99` · `--jinja --reasoning auto --reasoning-format deepseek` · `--metrics` · `--mmproj` · `--log-verbosity 4` · `--presence-penalty 1.0` · **`--spec-type draft-mtp`** with `--spec-draft-n-max 2 --spec-draft-n-min 0 --spec-draft-p-min 0` |
**Why this shape.** `general.architecture qwen35`: 33 blocks (32 trunk + `blk.32` = the MTP block),
**Why this shape.** `general.architecture qwen35`: 33 blocks (32 trunk + `blk.32` = the MTP block)
`full_attention_interval 4` ⇒ 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, `nextn_predict_layers 1`
(the head **is in the file**), trained window 262,144, `general.sampling.*` 1.0/0.95/20. Only the 8 full layers bill
per token and the DeltaNet layers add ~50 MiB of recurrent state **per slot**, so the shape is a single budget
@@ -44,7 +45,7 @@ projector, buffers and states, inside the 24,123 MiB the driver leaves addressab
18,816 MiB here) buys neither the third conversation nor the drafter; the fully quantized pair (17,408 B/token)
buys both and costs the keys their precision.
**MTP is ON (Frank, 2026-10-19).** The drafter was always good — 2026-10-18 measured acceptance 0.798 / 0.617 per
**MTP is ON (Frank, 2026-10-19).** The drafter was always good — 2026-10-18 measured acceptance 0.798 / 0.617
position, mean accepted length 2.41, and the publisher's card says depth 2 works best — and arming it is a
budget decision, not a flag: the draft context is built against the *target's* pool, so `blk.32` bills 4,096
B/position (2,352.00 MiB at 602,112) plus 480.97 MiB of its own CUDA0 buffers, and it triples the recurrent
@@ -58,7 +59,7 @@ and `UBATCH_SIZE=2048` beside an armed drafter is still the shape that halved de
**Harness contract** (`~/.omp/agent/models.yml`, provider `llamacpp (dankclaw-3090)`, id `frank-zealot`):
`contextWindow: 200704` — **the per-slot window**, because llama.cpp divides `--ctx-size` across slots and answers a
longer prompt with a 400, never a truncation — plus `maxTokens: 16384`, `input: [text, image]`,
longer prompt with a 400, never a truncation — plus `maxTokens: 16384`, `input: [text, image]`
`compat.reasoningContentField: reasoning_content`, `compat.reasoningEffortMap` (**load-bearing**: the template takes
only `xhigh|medium|low` and raises on anything else), `extraBody` 1.0 / 0.95 / 20 / `min_p 0`, and the
`compactionModel` of [[frank-archon]], [[frank-dragoon]] **and** frank-zealot all pointing here.
@@ -103,7 +104,7 @@ with 448 completion tokens (at `max_tokens: 64` the same request returns EMPTY c
| Knob | Effect from the shipped shape |
|---|---|
| `SPEC_TYPE=none` | the drafter off: hands back its 2,352.00 MiB of cache, 480.97 MiB of CUDA0 buffers and the 301.50 MiB of rollback planes it adds to the recurrent state (~3.1 GiB), at the cost of the ~+28 % decode the 2026-10-18 census measured whenever the card has room |
| `UBATCH_SIZE=2048` | +4.6 % prefill and only ~406 MiB of compute buffers at THIS shape (the ~1,306 MiB the 786,432-pool census showed does not transfer) — and the rung that HALVED decode beside an armed drafter (167.2 → 109.2 t/s) |
| `UBATCH_SIZE=2048` | — |
| `CTX_SIZE=401408` | two 200,704 conversations: frees 4,802 MiB of cache — `contextWindow` MUST follow the per-slot number |
| `CACHE_TYPE_V=f16` | exact V: +4,410 MiB at this pool with the drafter armed, which is a `cudaMalloc` death the seam refuses — legal only once the drafter or a slot is gone |
| `CACHE_TYPE_K=q8_0` | the fully quantized pair (17,408 B/token): frees 4,410 MiB and moves the quantization error into the keys |
@@ -112,23 +113,19 @@ with 448 completion tokens (at `max_tokens: 64` the same request returns EMPTY c
| Window | Payload | Runtime | Shape | Replaced because | Rollback |
|---|---|---|---|---|---|
| 2026-10-18 21:06 → 2026-10-19 | the same 9B MTP file and projector | PrismML fork 10777 | 2 × 262,144 (pool 524,288), exact f16 K+V = 16,384.00 MiB, `SPEC_TYPE=none`, batch 4096 / ubatch 2048; boot ledger KV 16,384.00 / RS 100.50 / compute 1,408.06 + 1,088.07 CUDA_Host / mmproj worst case 868.32 charged after the fit check, settled **21,176 used / 2,949 free**, `n_ctx_slot = 262144`, within 24 MiB of the projection; floors 134.1–141.5 t/s single-stream, 256.6–265.1 aggregate over two, 3,159.5 t/s prefill at 106,282, 71.5 t/s decode at 106k depth (contended — floors, not the shape's rates) | Frank ordered the third conversation back AND the drafter armed, and exact V funds neither: it bills 18,816 MiB at the pool that pays for the third slot, leaving nothing for the draft's 2,352 | one line each — `CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=f16 BATCH_SIZE=4096 UBATCH_SIZE=2048 SPEC_TYPE=none` — and redeploy; no payload move |
| 2026-10-18 13:19 → same evening (the boot `68afc6d` replaced) | the same 9B MTP file and projector | PrismML fork 10777 | 3 × 262,144 (pool 786,432), q8_0 K+V = 13,056 MiB, settled 17,956 / 6,169 MiB; 168.7 t/s decode (three runs 168.4–168.7), ~300 t/s over three streams (102.8 each), 2,877 t/s prefill at 106,418 | Frank ordered unquantized KV, and that pool cannot survive it: 24,576 MiB of f16 cache > 24,123 addressable | one env line — `CTX_SIZE=786432 PARALLEL=3 CACHE_TYPE_K/V=q8_0` — and redeploy; no payload move |
| 2026-10-18, 24 min (arm under the unit) | `Bonsai2-PTQ1_0/Ternary-Bonsai-2-27B-PTQ1_0.gguf` 5,946,648,928 B + hardlinked Q8 mmproj | PrismML fork 10777 | 2 × 200,704, batch/ubatch 3072/1536 | Frank scratched the 27B plan — ~48 t/s decode vs 76.6 for PQ2_0 on the same weights | `/home/dank/models/Bonsai2-PTQ1_0/` + `scripts/download-frank-zealot-bonsai2-ptq1.sh` |
| 2026-10-17 → 10-18 | Ternary-Bonsai-2-**27B** PQ2_0 7,206,168,928 B `3907dc16…` + Q8_0 mmproj 629,246,976 B | PrismML fork **10754** (`~/GitPackages/llama-prism/`) | 2 × 200,704 (pool 401,408), q8_0 K+V = 13,328 MiB, settled 22,664 / 1,461; 76.6 t/s decode, 109.4 aggregate over two streams, 691.7 t/s prefill at 166,027; MTP impossible — no head in the file | the 9B bills half the KV per token (8 full layers, not 16), which funded three slots at the native window and beat it on every rate | `/home/dank/models/Bonsai2-PQ2_0/` + the 10754 fork tree, `scripts/download-frank-zealot-bonsai2.sh`, revert the swap commit |
| 2026-10-18 21:06 → 2026-10-19 | the same 9B MTP file and projector | PrismML fork 10777 | — | Frank ordered the third conversation back AND the drafter armed, and exact V funds neither: it bills 18,816 MiB at the pool that pays for the third slot, leaving nothing for the draft's 2,352 | one line each — `CTX_SIZE=524288 PARALLEL=2 CACHE_TYPE_V=f16 BATCH_SIZE=4096 UBATCH_SIZE=2048 SPEC_TYPE=none` — and redeploy; no payload move |
| 2026-10-18 13:19 → same evening (the boot `68afc6d` replaced) | the same 9B MTP file and projector | PrismML fork 10777 | — | Frank ordered unquantized KV, and that pool cannot survive it: 24,576 MiB of f16 cache > 24,123 addressable | one env line — `CTX_SIZE=786432 PARALLEL=3 CACHE_TYPE_K/V=q8_0` — and redeploy; no payload move |
| 2026-10-18 03:14 → 13:19 (arm under the unit, then the leftover drop-in) | `Bonsai2-PTQ1_0/Ternary-Bonsai-2-27B-PTQ1_0.gguf` 5,946,648,928 B + hardlinked Q8 mmproj | PrismML fork 10777 | arm: 2 × 200,704, batch/ubatch 3072/1536. The leftover `frank-zealot.service.d/ptq1.conf` then made the 13:16 cutover deploy boot THIS payload at 3 × 133,888 (`new slot, n_ctx = 133888` x3, journal read 2026-10-19) until the 13:19 boot | — | `/home/dank/models/Bonsai2-PTQ1_0/` + `scripts/download-frank-zealot-bonsai2-ptq1.sh` |
| 2026-10-17 → 10-18 | Ternary-Bonsai-2-**27B** PQ2_0 7,206,168,928 B `3907dc16…` + Q8_0 mmproj 629,246,976 B | PrismML fork **10754** (`~/GitPackages/llama-prism/`) | — | the 9B bills half the KV per token (8 full layers, not 16), which funded three slots at the native window and beat it on every rate | `/home/dank/models/Bonsai2-PQ2_0/` + the 10754 fork tree, `scripts/download-frank-zealot-bonsai2.sh`, revert the swap commit |
| 2026-10-07 → 10-17 | Qwen3.5-9B-Uncensored-HauhauCS-Aggressive Q6_K 7,359,259,008 B `c0ba7beb…` + BF16 mmproj 921,704,448 B | llama.cpp master `5fc4f3c` CUDA sm_86 (`~/GitPackages/llama-master/build-cuda/`) | 2 × 200,704, exact f16 K+V, MTP off (no `nextn` tensors) | the ternary 27B bought a larger model class at the same window | `/home/dank/models/Qwen35-9B-HauhauCS-Q6K/` + the master build, `scripts/download-frank-zealot.sh` |
| 2026-10-07 (same day) | Gemma4-26B-A4B-QAT Q4_K_M 16,796,015,520 B + BF16 mmproj 1,194,827,776 B + unused `gemma4-assistant` MTP 251,937,728 B | llama.cpp master CUDA | 1 × 200,704, f16 K + q8_0 V, ubatch 1024, MTP off; 21,868 / 2,257 settled, 139.9 t/s decode (the ~4 B-active MoE is at home here); MTP ON would cost ~1.9 GiB and boot at 319 MiB free | ONE slot serialized compaction; the 9B's lighter weights bought two slots and exact KV | `/home/dank/models/Gemma4-26B-A4B/` (ungated — plain `curl -fL`) |
| 2026-10-03 → 10-05 | MiMo-V2.6-Distill-Qwen-9B Q6_K 7,793,710,624 B `ef96d05a…` + f16 mmproj 918,166,048 B | llama.cpp master CUDA | 2 × 200,704, f16 K+V, batch 4096 / ubatch 2048; 21,846 / 2,279 settled, 94.44 t/s decode | Frank ordered the uncensored 27B with MTP drafted | `/home/dank/models/MiMo-Q6K/` |
| 2026-10-07 (same day) | Gemma4-26B-A4B-QAT Q4_K_M 16,796,015,520 B + BF16 mmproj 1,194,827,776 B + unused `gemma4-assistant` MTP 251,937,728 B | llama.cpp master CUDA | — | ONE slot serialized compaction; the 9B's lighter weights bought two slots and exact KV | `/home/dank/models/Gemma4-26B-A4B/` (ungated — plain `curl -fL`) |
| 2026-10-03 → 10-05 | MiMo-V2.6-Distill-Qwen-9B Q6_K 7,793,710,624 B `ef96d05a…` + f16 mmproj 918,166,048 B | llama.cpp master CUDA | — | Frank ordered the uncensored 27B with MTP drafted | `/home/dank/models/MiMo-Q6K/` |
| 2026-10-05 (committed `77decb5`→`380aafa`, **never deployed**) | Ornith-1.5-9B Q8_0 | llama.cpp master CUDA | 1 × 262,144, f16 K+V | Swift cut over the same evening while this deploy sat blocked at the whole-tree docs gate | `/home/dank/models/Ornith-Q8/`; revert to `380aafa`'s tree |
| 2026-10-05 → 10-07 | Swift-1.5-27B-Uncensored RCO-IQ3_XXS-mtp 10,442,827,392 B `e13d3e74…` + publisher `mmproj-BF16.gguf` 931,145,888 B | llama.cpp master CUDA | 2 × 114,688, q8_0 K+V, `draft-mtp` depth 3, ubatch 2048; 23,120 / 1,005 settled (489 MiB at the 2 × 131,072 shape — the tightest fit ever here), 63.3 t/s at the recipe temp, 72.9 at temp 0, 48.0 with the drafter off | the 9B Q6_K rung returned the 200k window and exact KV | `/home/dank/models/Swift15-UncIQ3XXS/`; the IQ3_S-mtp rung at `/home/dank/models/Swift15-IQ3S/`; commits `da36fa3` → `f2575a1` → `d8da7a9` |
| 2026-10-02 → 10-03 | MiMo Q5_K_M 6,876,124,704 B `04cf46f2…` + the same f16 mmproj (hardlinked) | llama.cpp master CUDA | 2 × 200,704, f16 K+V, batch 2048 / ubatch 1024; 20,530 used / 4,046 apparent (true `free` 3,571), 104.84 t/s decode | the ~3.5 GiB it left free bought the Q6_K rung | `/home/dank/models/MiMo-Q5KM/` |
| 2026-10-05 → 10-07 | Swift-1.5-27B-Uncensored RCO-IQ3_XXS-mtp 10,442,827,392 B `e13d3e74…` + publisher `mmproj-BF16.gguf` 931,145,888 B | llama.cpp master CUDA | — | the 9B Q6_K rung returned the 200k window and exact KV | `/home/dank/models/Swift15-UncIQ3XXS/`; the IQ3_S-mtp rung at `/home/dank/models/Swift15-IQ3S/`; commits `da36fa3` → `f2575a1` → `d8da7a9` |
| 2026-10-02 → 10-03 | MiMo Q5_K_M 6,876,124,704 B `04cf46f2…` + the same f16 mmproj (hardlinked) | llama.cpp master CUDA | — | the ~3.5 GiB it left free bought the Q6_K rung | `/home/dank/models/MiMo-Q5KM/` |
| 2026-10-01 → 10-02 | MiMo Q6_K (same file as above) | llama.cpp master CUDA | 2 × 262,144, f16 K + q8_0 V | one night: the Q5 rung bought the 2–3 GiB headroom band back | `/home/dank/models/MiMo-Q6K/` (same dir) |
| 2026-10-01 (hours) | MiMo-V2.6 INT8-W8A16-AutoRound, 8 shards 12,891,718,440 B | SGLang in venv `~/GitPackages/sglang-cu128` (torch cu130 + an in-venv CUDA 13.0 nvcc) | 2 × 262,144 over a pinned 606,208 pool, `fp8_e4m3` KV, NEXTN off; 83.9 t/s decode, ~2,050–3,370 t/s eager prefill, ~60 s boot warm | llama.cpp was the only runtime exposing asymmetric per-tensor KV | payload purged — re-fetch `old/download-frank-zealot-sglang-int8.sh`; the venv is still on disk |
| 2026-09-17 → 10-01 | Gemma4-12B-QAT-Uncensored Q4_K_M 6.87 GiB + BF16 mmproj (it carried an **audio** encoder) + `mtp-gemma-4-12B-it.gguf` sidecar 0.24 GiB | llama.cpp master `37b53fd` CUDA with `-DGGML_CUDA_FA_ALL_QUANTS=ON` (`~/GitPackages/llama.cpp-mainline/build-cuda/`) | 4 × 262,144, q8_0 K+V, `draft-mtp,ngram-mod` depth 3 / poll 50; idle 22,346–23,300 MiB, ~965 t/s prefill and 257–302 t/s decode at 254k depth; each extra full-context slot costs 3,450 MiB so 4 is the max | the MiMo GGUF path (qwen35 hybrid, asymmetric KV) took the card | re-fetch `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced`; era trio in `git show 374041c:scripts/frank-small.service` / `.env` / `serve-frank-small.sh` |
| 2026-09-19 (same-day revert) | `gemma-4-12B-it-qat-w4a16-ct` 10,264,229,896 B + **bf16** assistant 845,719,296 B | vLLM dev image `localhost/vllm/vllm-openai:frank-small` (`8dad46d79c46`, still in rootless storage), rootless Quadlet | 262,144 window, bf16 KV pool 289,647 tokens, 2 concurrent sessions (the 3rd queues), image + audio; 1,293 t/s prefill @67k falling to 608 @213k, ~13.6 t/s decode @213k depth, MTP acceptance 2.80–3.56, boot 135 s | reverted to llama.cpp the same day | both payloads purged; `git show 374041c^:dankclaw/frank-small/Dockerfile` and `…:dankclaw/frank-small/quadlet/frank-small.container` |
| 2026-09-16 → 09-17 | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S + BF16 projector, in-file MTP drafted | mainline llama.cpp | single-slot 3090 shape | superseded by the Gemma4-12B payload | re-download from ISTA-DASLab — no payload and no fetch script survive |
| 2026-09-02 → 09-15 (**Dankserver**, `:7770`, the card's previous host) | `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` ~35 GB (before it official `Qwen3.6-35B-A3B-FP8`) | FreeToken v0.1.2 (`ft`), venv `~/freetoken/.venv` | MoE cache 2,048 slots + KV 453,005 tokens + 24 mamba-state slots; ~30 t/s single-stream decode, ~87 t/s at batch 4, ~1,130 t/s prefill at 4k | the 3090 left Dankserver and `frank-medium-rocmfpx` took the endpoint and port | host units `~/.AGENTTRASH/20260915-frank-zealot-retire/host-units/`, repo trio in `scripts/.AGENTTRASH/` |
| ~2026-08 (Dankserver 3090, pre-FreeToken) | Qwen3.6-35B-A3B Q6_K_P, then Tiel-Coder-35B-A3B UD-Q6_K_XL, with an Ornith Q6_K rung | llama.cpp `555881ebc` (b10121) CUDA | `--n-cpu-moe 38` deployed (36 recommended), batch 4096 / ubatch 2048, 700,416-token f16 K+V pool (800,768 cells did not fit); ~45.5 t/s decode and 1,188 t/s prefill at `ncmoe 35` | FreeToken kept experts in host RAM instead of CPU-offloading layers | re-download; nothing from this rung survives on disk |
| 2026-10-01 (hours) | MiMo-V2.6 INT8-W8A16-AutoRound, 8 shards 12,891,718,440 B | SGLang in venv `~/GitPackages/sglang-cu128` (torch cu130 + an in-venv CUDA 13.0 nvcc) | — | llama.cpp was the only runtime exposing asymmetric per-tensor KV | payload purged — re-fetch `old/download-frank-zealot-sglang-int8.sh`; the venv is still on disk |
| ≤ 2026-09 — 6 earlier eras: payloads, shapes and every measured number destroyed 2026-10-19 as obsolete | — | — | — | — | —
## Laws learned here
@@ -145,10 +142,12 @@ with 448 completion tokens (at `max_tokens: 64` the same request returns EMPTY c
3 cells, and an armed depth-2 drafter triples it to 452.25), compute buffers at graph capture, the mmproj after the
fit. On gemma4-class hybrids `attention_k_eq_v` does **not** shrink the allocation — the cache reserves the K/V pair
twice, 6.125 B/elem at f16 K + q8_0 V.
- **`--flash-attn on` is pinned, not decorative**: a quantized V exists only on the FA path and the engine falls back silently without it. llama.cpp has **no true fp8 KV dtype** — q8_0 is the 8-bit class (1.0625 B/elem, ~8.5 effective bits). `GGML_CUDA_FA_ALL_QUANTS` is vestigial and build-specific: the PrismML fork carries it ON (since 2026-09-17) while the master build had it OFF and still dispatched Q4/Q5 KV unconditionally (`ggml-cuda/fattn.cu` has no guard), so quantized KV never required it.
- **`--flash-attn on` is pinned, not decorative**: a quantized V exists only on the FA path and the engine falls back silently without it. llama.cpp has **no true fp8 KV dtype** — q8_0 is the 8-bit class (1.0625 B/elem, ~8.5 effective bits). `GGML_CUDA_FA_ALL_QUANTS` is **deprecated upstream and inert in both trees on this host** (no reference anywhere in `ggml/src/ggml-cuda/`, read 2026-10-19): master's `ggml/cmake/common.cmake` warns `use GGML_CUDA_FA_QUANTS=all instead`, and the successor's default — `GGML_CUDA_FA_QUANTS="q4_0-q4_0;q8_0-q8_0;f16-f16;bf16-bf16"` (`ggml/CMakeLists.txt:208`) — already compiles the quantized-KV kernels with the old flag OFF (master's `build-cuda` cache says `FA_ALL_QUANTS:BOOL=OFF` and it dispatched q8_0/q4_0 KV regardless). Prove a build's quantized-KV ability from its `CMakeCache.txt` `GGML_CUDA_FA_QUANTS` line, never from the retired flag or the build command you remember.
- **Prices come from boots, not formulas, and an arm belongs to the payload it was measured on**: 3 × 180,224 +
q8_0 V fits with Q5_K_M (21,684 used / 2,441 free) and does **not** fit with Q6_K inside the 2–3 GiB band. The
mmproj worst case (~870 MiB) is printed before fitting but never billed by the engine's own fit check. Only
q8_0 V fits with Q5_K_M (21,684 used / 2,441 free) and does **not** fit with Q6_K inside the 2–3 GiB band.
The mmproj worst case is printed before fitting (`estimated worst-case memory usage of mmproj is
873.10 MiB` on the Bonsai-2 27B boot) but never billed by the engine's own fit check: that boot
projected **21,439 MiB** and settled at **22,664**. Only
`tests/frank_zealot_unit.bats`'s fit gate can see a boot that would die at `cudaMalloc`, so every retune keeps it
green — and the first verbosity-4 ledger plus settled `nvidia-smi` of a new dtype belongs on this page, because
projections drift.
@@ -195,11 +194,14 @@ with 448 completion tokens (at `max_tokens: 64` the same request returns EMPTY c
**500 "The model produced output that does not match the expected Content-only format"**, because build 10777
guards the raw path the moment reasoning markup leaks (older builds only logged `failed to parse reasoning
content` and answered anyway). The honest per-request instrument here is **`/v1/chat/completions`**, whose
non-stream response carries the same `timings` object (`prompt_n`, `prompt_per_second`, `predicted_n`,
non-stream response carries the same `timings` object (`prompt_n`, `prompt_per_second`, `predicted_n`
`predicted_per_second`). Keep `VERBOSITY=4` — 3 is the engine default and cannot prove the KV or draft sizes,
while levels above 4 flood the journal.
- llama.cpp does **not** validate the requested model id: it answers and echoes the served `ALIAS`, so a rename breaks id-keyed consumers (the OpenWebUI `model` row, its `access_grant`, `openai.api_configs`) and nothing server-side. Alias history: `frank-small` → `frank-secondary` (2026-10-06) → `frank-zealot` (`7c77326`, 2026-10-09); the service and unit took the name 2026-10-10 (`55f87fc`).
- The first request after a boot is a warmup artifact (37 t/s class) and is never a measurement; never time a wall-clock aggregate on a generation that stops on its own (use `ignore_eos`).
- The first request after a boot is a warmup artifact, never a measurement: on the Bonsai-2 27B
boot request #1 gave 2.6 t/s prefill / 37.7 t/s decode and request #2 gave 1,258 t/s prefill /
54.7 t/s per stream (two concurrent, the 109.4 aggregate in the Eras row) — a 3x spread on one
shape. Never time a wall-clock aggregate on a generation that stops on its own (use `ignore_eos`).
## Files
@@ -213,7 +215,7 @@ with 448 completion tokens (at `max_tokens: 64` the same request returns EMPTY c
- `scripts/download-frank-zealot-bonsai9.sh` (active) · `…-bonsai2.sh` · `…-bonsai2-ptq1.sh` ·
`scripts/download-frank-zealot.sh` (master-era GGUFs) · `old/download-frank-zealot-sglang-int8.sh` (SGLang).
- `tests/frank_zealot_unit.bats` — stub-argv seam plus the VRAM fit gate.
- `scripts/deploy-remote.sh` — `deploy_llm_zealot` (payload-path gates, MainPID-hardened health); see [[deployment]].
- `shared/prometheus/prometheus.yml`, `shared/prometheus/rules/llm-fleet.yml`,
- `scripts/deploy-remote.sh` — `deploy_llm_zealot` (payload-path gates, MainPID-hardened health);
- `shared/prometheus/prometheus.yml`, `shared/prometheus/rules/llm-fleet.yml`
`shared/grafana/provisioning/dashboards/llm-fleet.json` — the scrape, the alert, the dashboard.
- Live state, not repo files: the OpenWebUI picker row (back up `webui.db` before re-keying) and `~/.omp/agent/models.yml`.
serverAppConfigs-docs/LLM/freetoken-benchmark-sweep.md
-94
View File
@@ -1,94 +0,0 @@
---
type: research
title: FreeToken Benchmark Sweep — Dankserver RTX 3090
description: KV pool vs decode speed sweep across bs1/bs3/bs4/bs6 on Dankserver with Qwen3.6-35B-A3B-FP8 via FreeToken v0.1.2.
tags:
- freetoken
- benchmark
- dankserver
- frank-zealot
- qwen3.6
timestamp: 2026-09-02
---
# FreeToken Benchmark Sweep — Dankserver RTX 3090
Full sweep of KV pool size vs decode speed at batch sizes 1, 3, 4, and 6.
Measured 2026-09-02 with FreeToken v0.1.2 serving `Qwen/Qwen3.6-35B-A3B-FP8`
on Dankserver (RTX 3090 24 GB, 60 GB RAM, driver 595.84).
## Methodology
- Each point: restart at low KV (8k), live-rebuild upward via
`ft ctl cache --kv N --moe M` (auto-computed MoE slot count).
- Warmup: 2–3 iterations at 64 tokens (fills LRU expert cache).
- Measured: 2–3 iterations at 128 tokens, `temperature=0`, `ignore_eos=true`.
- Prompt: "Count from 1 to 100." (short — isolates decode bandwidth).
- Per-token KV cost: ~20 KiB (10 full-attention layers × 2 KV heads ×
256 dim × bf16). Per-slot MoE cost: ~3 MiB.
- GPU budget: ~17.6 GiB after weights + fixed caches.
## bs1 vs bs3 vs bs4 vs bs6
| KV pool | Residency | bs1 | bs3 agg | bs3/req | bs4 agg | bs4/req | bs6 agg | bs6/req |
|---------|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| 8,192 | ~89% | 56.7 | 108.8 | 36.3 | 150.6 | 37.7 | 132.1 | 22.0 |
| 65,536 | ~83% | ~52 | 106.1 | 35.4 | 148.0 | 37.0 | 131.8 | 22.0 |
| 131,072 | ~76% | ~48 | 99.9 | 33.3 | 144.4 | 36.1 | 131.8 | 22.0 |
| 262,144 | ~62% | ~48 | 85.3 | 28.4 | 131.3 | 32.8 | 130.5 | 21.8 |
| 400,000 | ~47% | ~42 | 73.3 | 24.4 | 116.5 | 29.1 | 161.1 | 26.9 |
| 524,288 | ~30% | 34.8 | 59.8 | 19.9 | 102.2 | 25.5 | 150.6 | 25.1 |
| 655,360 | ~19% | 30.4 | 42.8 | 14.3 | 86.9 | 21.7 | 131.5 | 21.9 |
| 700,000 | ~15% | 28.3 | 38.5 | 12.8 | 83.0 | 20.8 | 121.1 | 20.2 |
## Key Findings
### No cliff — gentle slope
The LRU expert cache absorbs residency reductions gracefully. Even at
5% residency (544 slots), bs1 decode is 25.2 tok/s — 44% of baseline.
### Free zone up to ~262k
Decode speed is flat from 8k through 262k at bs1 and bs4. Giving up
expert residency (89% → 62%) costs nothing because the LRU working set
fits in fewer slots.
### bs4 is the sweet spot
bs4 dominates at every high-residency operating point. Per-request
speed at bs4 (37.7 tok/s at 8k) is *faster* than bs3 (36.3) because 4
concurrent requests better fill the PCIe expert-streaming pipeline.
### bs6 crossover at ~400k
At high residency, bs6 aggregate (132) < bs4 aggregate (150) — the
bottleneck is GPU compute on attention layers, not PCIe. At low
residency (400k+), the bottleneck flips to PCIe expert streaming and
bs6 wins (161 vs 116 at 400k) because more concurrency fills the
pipeline.
### bs3 is strictly worse than bs4
Fewer concurrent users leave PCIe idle gaps. At every point past 8k,
bs3 per-request is slower than bs4 per-request. Don't run bs3 — set
`--max-running-requests 4` regardless of actual user count.
### 800k doesn't fit
4 × 200k = 800k exceeds the GPU budget even at minimum MoE (256 slots).
The absolute ceiling is ~750k tokens. With radix prefix sharing,
effective capacity exceeds naive `pool ÷ concurrency`.
## Comparison: old llama.cpp service
The old Tiel-Coder llama.cpp service ran 200,700 total context across
3 parallel slots (66,900/slot) at ~99 tok/s aggregate at concurrency 4.
FreeToken at 131k KV (bs4) does 144 tok/s aggregate — 45% faster — with
the caveat that it serves base Qwen3.6-35B-A3B weights, not the
Tiel-Coder finetune.
## See Also
- [[frank-zealot]] — production runbook
- [[dealign-ornith-crack]] — the prior llama.cpp deployment (retired)
serverAppConfigs-docs/LLM/ikllama-ornith-dankserver.md
-262
View File
@@ -1,262 +0,0 @@
---
type: reference
title: ik_llama.cpp on Dankserver (Ornith Q8_0)
description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-zealot on the RTX 3090. Q8_0 model, KV K default f16 / V q6_0 with Hadamard on V only (iq5_k is not a KV-cache type in this build; q8_KV crashes on this arch), split-mode none, speculative decoding disabled, 3 parallel slots x 163840 ctx, the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
tags:
- ikllama
- llamacpp
- cuda
- dankserver
- moe
- ornith
- kv-cache
- hadamard
timestamp: 2026-08-26
---
# ik_llama.cpp on Dankserver (Ornith-1.5-35B-A3B CRACK)
> **Active 2026-08-26.** `frank-zealot` switched from upstream
> llama.cpp (b10121) to `ikawrakow/ik_llama.cpp` (commit `b166e269`) on the
> RTX 3090. This page documents the build, the mandatory FA build flags, and
> the MoE / CPU-offload runtime arguments.
The model is `dealignai/Ornith-1.5-35B-A3B-UNCENSORED` (CRACK **Q8_0**), 35B-total /
3B-active MoE-dense hybrid with an MTP head in the GGUF (mmproj f16 vision
encoder). Served as **three parallel 160000-context slots** (`PARALLEL=3`).
Speculative decoding is **disabled**. See [[dealign-ornith-crack]] for the
model runbook.
## Why ik_llama.cpp
ik_llama.cpp is ikawrakow's llama.cpp fork with heavy CPU + CUDA
optimizations, IQK/AVX-512 kernels, `--fit` auto-offload, and fast q8_KV
cache. Dankserver is a CUDA (RTX 3090) + CPU-MoE-offload box, which is
exactly the supported hybrid the fork targets. Only CPU and CUDA backends
are fully functional; ROCm / Vulkan / Metal are unsupported upstream.
## Build recipe (CUDA 13.2 via staged pip toolkit)
The system CUDA toolkit on Dankserver is **removed** (`nvidia-cuda-toolkit`
is in `rc` state; no `/usr/local/cuda*`, no `/usr/bin/nvcc`). The CUDA
13.2 toolkit is installed via **pip** inside the vllm venv at
`/home/dank/vllm/.venv/lib/python3.12/site-packages/nvidia/cu13/`. A staged,
self-contained toolchain at `/home/dank/cuda13-stage` adds the unversioned
`.so` aliases and static libs CMake needs.
```bash
# 1. Clone a pinned ik_llama.cpp
git clone https://github.com/ikawrakow/ik_llama.cpp.git /home/dank/ik_llama.cpp
cd /home/dank/ik_llama.cpp
git checkout b166e269 # the version this deployment was built from
# 2. Configure (stage already symlinks nvcc/include/lib for cu13)
cmake -B build \
-DCMAKE_BUILD_TYPE=Release \
-DGGML_CUDA=ON \
-DGGML_CUDA_FA_ALL_QUANTS=ON \
-DGGML_IQK_FA_ALL_QUANTS=ON \
-DCMAKE_CUDA_ARCHITECTURES=86 \
-DCMAKE_CUDA_COMPILER=/home/dank/cuda13-stage/bin/nvcc \
-DCUDAToolkit_ROOT=/home/dank/cuda13-stage \
-DGGML_NATIVE=ON
# 3. Build
cmake --build build --config Release -j 8
```
### ⚠️ MANDATORY: always include the FA all-quants flags
> **Rule.** Any ik_llama.cpp build that serves a quantized KV cache
> (q8_KV / q8_0 / Hadamard / any non-f16 KV) **MUST** be compiled with
> BOTH:
> - `-DGGML_CUDA_FA_ALL_QUANTS=ON` — compiles every K/V cache quant combo
> into the **CUDA** FlashAttention kernels. Without it, CUDA FA only ships
> a subset and **errors at runtime** telling you to recompile when a combo
> (e.g. q8_KV + Hadamard V-cache) isn't present. The RTX 3090 keeps KV on
> the GPU, so this is the flag that matters here.
> - `-DGGML_IQK_FA_ALL_QUANTS=ON` — compiles all quant types (Q4_0/Q4_1/
> IQ4_NL on top of the always-on F16/Q8_0/Q6_0/BF16) into the **CPU/IQK**
> FA kernels. Relevant because the MoE experts run on the CPU; the KV
> cache that backs CPU-side attention uses the IQK FA path.
>
> These are the old "asymmetrical quants" compile-time gates. On a CUDA
> host, omitting `GGML_CUDA_FA_ALL_QUANTS` is the #1 way to get a
> "recompile with GGML_CUDA_FA_ALL_QUANTS" crash at load time.
`GGML_NATIVE=ON` lets GCC target the host (the Ryzen 7 7700X is Zen4, so
`-march=native` enables the AVX512 / VNNI IQK kernels automatically —
verifiable with `system_info: AVX512_VNNI = 1` in the boot log).
### CUDA stage note
`/home/dank/cuda13-stage` is the pip `cu13` toolkit plus the symlinks CMake's
(CUDAToolkit) finder needs (`libcudart.so`, `libcublas.so`, `libcublasLt.so`
and static `libcudadevrt.a`/`libcudart_static.a`). It also works around the
CCCL CUDA version skew by defining
`-DCCCL_DISABLE_CTK_COMPATIBILITY_CHECK` in `CMAKE_CUDA_FLAGS` (the pip
package ships nvcc 13.2.78 but reports `CUDART_VERSION 13000`, which trips
a strict header lint). Do not re-run configure without that flag or you get
a hard compile error.
## Runtime configuration
Files: `scripts/serve-frank-zealot.sh`, `scripts/frank-zealot.env`,
`scripts/frank-zealot.service` in the repo. The service just calls
the serve script; the env file holds all tunables.
### KV cache: K default f16 / V q6_0 + Hadamard (V only)
- `CACHE_TYPE_K=f16` — K cache at the default full-precision f16
(passes `--cache-type-k f16`). No Hadamard on K.
- `CACHE_TYPE_V=q6_0` — V cache at 6-bit k-quant
(passes `--cache-type-v q6_0`).
- `V_CACHE_HADAMARD=1` — `--v-cache-hadamard` on V only. The Hadamard
transform preserves V quality at 6-bit; K is full-precision f16 and does
not need it (`K_CACHE_HADAMARD=0`). Implemented for both CUDA
(`ggml-cuda/hadamard.cu`) and CPU (IQK) paths. Flash-attention is **on
by default**.
> **KV cache type history on this build (`b166e269`):**
> - `iq5_k` is **not** a KV-cache type — the cache whitelist is
> `f32/f16/bf16/q8_0/q4_0/q4_1/iq4_nl/q5_0/q5_1/q6_0/q8_KV`; `iq5_*` only
> exist as weight quants. `--cache-type-k iq5_k` throws `Invalid cache
> type`.
> - `q8_KV` **crashes** with a `ggml view` size assert
> (`ggml.c:5427: data_size + view_offs > ggml_nbytes(view_src)`) on this
> QWEN3MOE/MTP + multimodal arch — independent of PARALLEL/speculation.
> - The deployment settled on K at default **f16** (full precision, no
> Hadamard) with V at **q6_0 + Hadamard**.
With 3 parallel slots at 163840 context the KV cache is large; keeping K at
full precision f16 while quantizing only V (with Hadamard) is the chosen
budget split. The GPU-side expert set stays reduced to fit (see the expert
section).
### Split mode: none (single GPU)
- `SPLIT_MODE=none` → `--split-mode none` (the default). Dankserver has a
single RTX 3090 yet ik_llama rejects `--split-mode graph` on a box with a
single device:
> `llama error loading model: It is not possible to use split mode 'graph'
> with less than 2 devices`
So `graph` is **not** a harmless no-op on one GPU — it is a hard load
error (this bit us on the first deploy and loop the service). It also still
works with the `--override-tensor` expert pins.
> **Lesson.** ik_llama `--split-mode graph` requires ≥2 devices. On a
> single-GPU host you must use `none` (or `layer`, which also needs >1).
> Do not assume `--split-mode graph` degrades gracefully on one GPU.
### Expert offload (MoE: CPU with a partial GPU slice)
The expert-offload pattern from upstream llama.cpp carries over: the
`--override-tensor` regex pins the sparse expert (`ffn_*_exps`) tensors to
the CPU buft, leaving the output-adjacent blocks 35-40 on CUDA. On the
Q8_0 model each ffn expert tensor is **272 MiB** (~4 tensor/block ≈ 1088
MiB/block). With `PARALLEL=3` the KV cache tripled, so most expert blocks
are pinned to CPU; only blocks **35-40** (output-adjacent, latency- and
quality-relevant) stay on CUDA. Full-attention blocks 3,7,11,15,19,23 have
no exps and are untouched by the regex.
- `OVERRIDE_EXPERT` — the layer-bounded regex listing the blocks whose
experts stay on CPU: `(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|
24|25|26|27|28|29|30|31|32|33|34)`. Blocks 35-40 (and the always-on
full-attention set) are left off the list so their experts load to CUDA.
- `OVERRIDE_BUFT=CPU`.
- `N_GPU_LAYERS=999` — ik_llama `-ngl` / `--gpu-layers` only accepts an
integer (the literal `all` is rejected). 999 means "load as many tensor
layers as possible", while the expert override still pins the listed exps
to CPU.
> **Fused MoE** (`fused_moe`, `fused_up_gate`, `fused_mmad`) is **on by
> default** in ik_llama — there is no positive flag; only `-no-fmoe`
> disables it. `only_active_experts` is also on by default. Confirmed in the
> boot log for the running service.
To move more experts back to the GPU later, drop more block numbers from
`OVERRIDE_EXPERT` (each block is ~1088 MiB on Q8_0; verify `nvidia-smi`
headroom first and keep ~2-3 GiB margin for KV/compute). With the 3-slot KV
cache there is little headroom, so the default keeps only 35-40 on GPU.
### Qwen-Sharp chat template via --jinja
`--chat-template-file` alone only stores the template text; ik_llama **does
not** render a custom template as real Jinja unless `--jinja` is also given
(`inputs.use_jinja ? common_chat_templates_apply_jinja : <non-jinja
fallback>`). Without `--jinja` it warns it only supports "commonly used
templates" and silently uses built-in handlers, dropping the Qwen-Sharp
template's thinking / image / tool formatting. So the serve script passes
both `--chat-template-file` and `--jinja` (gated on `USE_JINJA`, set to `1`).
### Speculative decoding: disabled (3 parallel slots)
`SPEC_TYPE=none` (passed as `--spec-type none`). Speculative decoding is
intentionally disabled because ik_llama **limits any speculative stage to a
a single server slot**: with `--parallel 3` a configured spec stage
hard-exits at load with
> `ERR [load_model] Speculative decoding is currently limited to a single
> server slot (-np 1).` / `n_parallel=3`
This service keeps **3 parallel slots** (one per user family via
`PARALLEL=3`), so speculative decoding cannot be used here. MTP stays
disabled. The model is a low-parameter MoE (3B active of 35B) and does not
need the speculation it would cost a slot for.
> **Lesson.** Do not combine speculative decoding with `--parallel > 1` in
> ik_llama. Either drop to `-np 1` (one slot) and keep a spec stage, or keep
> multiple slots and set `SPEC_TYPE=none`. For a multi-user service the
> extra slots win.
### Flash-attention + sampling
Flash-attention is **on by default** in ik_llama (`flash_attn=true`), so no
`-fa`/`--flash-attn` flag is needed. Sampling, thread, batch, context, and
checkpoint tunables (`TEMPERATURE`, `TOP_K`, `TOP_P`, `MIN_P`, thread,
`BATCH_SIZE`, `UBATCH_SIZE`, `CTX_SIZE`, `PARALLEL`, `CTX_CHECKPOINTS`,
`NO_MMAP`) are unchanged and pass straight through.
## MoE / hybrid-arg reference (not in upstream llama.cpp)
Several ik_llama knobs matter for a CPU-MoE offload host:
| flag | meaning | our use |
|------|---------|---------|
| `--split-mode graph` | split tensors+graph across devices; strong for MoE; **requires 2+ devices** | single 3090 → use `none`
| `-ooae` / `--offload-only-active-experts` | default ON: copy **only active** experts RAM→VRAM | keep ON (huge for A3B: few experts active) |
| `-grt q8_0` | reduce type for inter-GPU data exchange | not needed (single GPU) |
| `--fit` | auto-offload to fit VRAM | conflicts with `--override-tensor`; do not use together (runtime error) |
| `-n-cpu-moe N` / `--cpu-moe` | keep MoE weights of first N / all layers on CPU | also conflicts with `--fit`; our `--override-tensor` is finer-grained |
| `--fit-margin` | VRAM safety margin for `--fit` | only relevant with `--fit` |
The current deployment uses `--override-tensor` (not `--fit` or
`--n-cpu-moe`) for the expert offload, so the `--fit`-conflict rules do not
apply.
## Rollback
The previous runtime `upstream llama.cpp b10121` and its binary
(`/home/dank/llama.cpp/build/bin/llama-server`) are **left untouched**. To
rollback to the old stack, change `LLAMA_SERVER=` in
`scripts/frank-zealot.env` back to that path and
`CACHE_TYPE_K/V` back to `f16` with Hadamard off, and re-enable the ngram
spec / `--spec-type` flags. The model file is not changed either way.
## See Also
- [[dealign-ornith-crack]] — the Ornith Q6_K model run
- [[Dankserver System Specs]] — the host
- [[deployment]] — service-scoped git deploy
## Citations
- [ikawrakow/ik_llama.cpp README](https://github.com/ikawrakow/ik_llama.cpp) —
MoE offload guidance and the `--split-mode graph` + CPU-offload warning
- [ik_llama.cpp serverAppConfigs-docs/build.md](https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/build.md) —
build flags incl. `GGML_CUDA_FA_ALL_QUANTS` and `GGML_IQK_FA_ALL_QUANTS`
- [ik_llama.cpp serverAppConfigs-docs/parameters.md](https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/parameters.md) —
q8_KV (PR 208), K/V Hadamard options, `-cuda graphs=0` workaround, and
`--split-mode graph` MoE note (PR 1080)
serverAppConfigs-docs/LLM/index.md
+21 -61
View File
@@ -1,45 +1,27 @@
# LLM fleet — index
The three live endpoint pages are the only pages that carry a shape. Everything
else here is a technique, a piece of hardware, or a trial that taught a law — and
each of those owns exactly that one thing. The incidents behind the laws are in
[[log]].
else here is a technique or a piece of hardware, and each of those owns exactly
that one thing. There is no worklog and no era page: a payload swap is one row in
the endpoint's **Eras** table, and the numbers a retired era measured are gone.
## Live endpoints
| Page | Owns |
|---|---|
| [[frank-archon]] | The one frank-archon page: the live radiance shape and harness contract (read off scripts/frank-archon.env), the Eras… |
| [[frank-dragoon]] | The one runbook for frank-dragoon, the Strix Halo iGPU endpoint (192.168.0.143:7770): current shape from the env,… |
| [[frank-zealot]] | The one frank-zealot reference: the live shape on Dankclaw's RTX 3090 (:7771) exactly as scripts/frank-zealot.env… |
| [[frank-archon]] | The radiance endpoint: live shape and harness contract, every era with its rollback path, and the laws it taught. |
| [[frank-dragoon]] | The iGPU endpoint: live shape, every era with its rollback path, the laws it paid for, and the files that own it. |
| [[frank-zealot]] | The RTX 3090 endpoint: live shape exactly as the env ships it, every era with its rollback path, and its card laws. |
## Retired endpoints
| Page | Owns |
|---|---|
| [[frank-medium-batch-ubatch]] | llama-bench sweep of --batch-size against --ubatch-size on frank-medium (2x Radeon AI PRO R9700, layer split) at the… |
| [[frank-medium-llamacpp-dflash2]] | The former ACTIVE frank-medium deployment since 2026-09-16 (model swapped 2026-09-19) — mainline llama.cpp on… |
| [[frank-medium-model-swap]] | RETIRED-era swap procedure for the frank-medium endpoint (`:7770` is now [[frank-archon]]) — the… |
| [[frank-medium-radiance-mxfp4]] | RETIRED frank-medium era runbook — vllm-radiance serving hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 (ParoQuant MXFP6 W4A8,… |
| [[frank-medium-rocmfpx]] | charlie12345/ROCmFPX llama.cpp serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF as frank-medium on Dankserver… |
| [[frank-npu]] | The retired FastFlowLM endpoint on Dankclaw's XDNA2 NPU — what it served, why it died on 2026-09-11 with the… |
| [[frank-redo-2026-08-17]] | This page records the full redo of the inference fleet on 2026-08-17. |
| [[ikllama-ornith-dankserver]] | ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-zealot on the RTX 3090. Q8_0 model, KV K default f16 / V… |
| [[kingjones-q8agent-rocmfpx]] | **Status (historical, 2026-09-13):** was to deploy on Dankclaw. |
| [[qwen3.8-spec-matrix]] | This page records the speculative-decoding benchmark history and the |
## Pipeline Parallelism (series)
| Page | Owns |
|---|---|
| [[Ethernet Bandwidth Math]] | Bytes-per-token math for pipeline vs tensor parallelism, feasibility tables for 1/2.5/10 GbE, and real multi-node… |
| [[Expected Performance]] | Estimated tokens per second for a 25-40B model split across two nodes over 1/2.5/10 GbE, with real benchmark anchors. |
| [[llama.cpp RPC]] | llama.cpp distributed inference via ggml-rpc-server and the --rpc flag: architecture, CLI, versions, limitations, and… |
| [[Mixed-Vendor Experiments]] | Real-world NVIDIA + AMD Strix Halo multi-node experiments: what works (llama.cpp RPC, Vulkan split), what breaks… |
| [[Qwen3.6 27B Spec Methods]] | Qwen3.6-27B Q8 GGUF variants and llama.cpp speculative decoding (draft-mtp, draft-dflash, ngram-mod): flags, RPC… |
| [[Qwen3.8 PP Deployment]] | Split Qwen3.8-27B across the Strix Halo iGPU (head) and the RTX |
| [[Sample Script Set]] | An educational two-node llama.cpp RPC split of Qwen3.6-27B-Q8_K_P over ethernet. For reading only. Do not run or… |
| [[vLLM Pipeline Parallelism]] | vLLM multi-node pipeline parallelism: flags, Ray/mp backends, interconnect requirements, PP bubble, and PP +… |
## Techniques, hardware, trials
@@ -47,43 +29,21 @@ each of those owns exactly that one thing. The incidents behind the laws are in
|---|---|
| [[Dankclaw Improvements]] | This page collects concrete, sourced recommendations to improve Dankclaw |
| [[Dankclaw System Specs]] | Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference. |
| [[dankclaw-iommu-translation-off]] | Dankclaw boots iommu=pt since 2026-10-18 — identity DMA with the IOMMU armed, which restored the XDNA2 NPU at a… |
| [[Dankserver System Specs]] | Dankserver hardware and OS state since the 2026-09-20 board swap (ASUS ProArt B850-Creator WiFi NEO, 2x AMD Radeon AI… |
| [[dankserver-frontier-moe-feasibility]] | VERIFIED 2026-09-28 by measurement and source reading. Compares DeepSeek-V4.1-Flash, MiMo-V2.6-Flash-MOPD and… |
| [[dankserver-vram-growth]] | Investigation of the recurring cuBLAS crash on frank-zealot. What grows, what does not, and what to do next. |
| [[dealign-ornith-crack]] | dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with… |
| [[deployment]] | Scoped deployment and update process for the three inference services (frank-archon, frank-dragoon, frank-zealot). |
| [[dankclaw-iommu-translation-off]] | Why Dankclaw boots with identity DMA, what the earlier off-era bought and cost, and which tooling owns the boot line. |
| [[Dankserver System Specs]] | Dankserver hardware and OS state since the board swap: board, lane topology, cards, memory, kernel posture. |
| [[DFlash DSpark Speculative Decoding]] | DFlash and DSpark are the two newest speculative decoding methods. |
| [[Dual-R9700-Cooperation]] | Why frank-medium's two Radeon AI PRO R9700s could not run tensor-parallel on the old B650-E board (PCIe atomic… |
| [[FastFlowLM NPU Inference]] | FastFlowLM (FLM), the AMD NPU-first LLM runtime for Ryzen AI XDNA2 chips. Fit analysis for the 50 TOPS NPU on… |
| [[FreeToken Builds]] | Building FreeToken (the official franksdank fork of FreeTok Tokenworks' freetoken vLLM-MTP runtime) on Dankclaw — the… |
| [[freetoken-benchmark-sweep]] | KV pool vs decode speed sweep across bs1/bs3/bs4/bs6 on Dankserver with Qwen3.6-35B-A3B-FP8 via FreeToken v0.1.2. |
| [[GGUF Compatibility]] | How to answer "will engine X load this GGUF?" from the file's header alone, why a custom tensor type forces the… |
| [[Heterogeneous Split Triage]] | Triage before building when someone asks to split a model across unequal accelerators on one host (24 GB GDDR6X +… |
| [[Hybrid CPU GPU Processing Dankserver]] | Dankserver uses NVIDIA GPU with system RAM for hybrid processing. |
| [[ling-3.0-flash-trial]] | `bailingmoe3-support` (commit `db4480bc8`, build b10411) at `/home/dank/llama.cpp-ling`. |
| [[ling-3.0-flash-vllm-plan]] | This page was the plan for a vLLM-ROCm deployment of `jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native` (INT4 W4A16, ~62… |
| [[Llama Cpp Multi-Device Splits]] | What llama.cpp can and cannot express across devices, the no-collective layer-split path, split balancing by… |
| [[Llama Cpp Serving Sizing]] | Slot-pool sizing against a headroom band, per-device VRAM accounting, prompt-cache/state sizing and the QSA… |
| [[LLM Bench Windows and Measurement]] | Stopping-and-restoring benchmark windows on the fleet LLM hosts (llama-bench sweeps, HTTP param benches, multi-arm… |
| [[LLM Performance Diagnosis]] | Probing why a served LLM is slow — the flat-depth probe, the expert-offload duty-cycle arithmetic, kernel-time… |
| [[Model Swap and Rollback]] | The full model-swap discipline for the LLM fleet — fit preflight, fetch and payload verification, runtime build/flag… |
| [[PCIe Link and Atomics]] | Link probing (idle downshift, path ceilings, CUDA/HIP bandwidth probes), the streaming-vs-compute ceiling arithmetic,… |
| [[GGUF Compatibility]] | Answering whether an engine loads a GGUF from the file header alone, and pinning a fork build that can. |
| [[Heterogeneous Split Triage]] | Triage before splitting across unequal accelerators: link characterisation, real bottleneck, what placement never buys. |
| [[Llama Cpp Serving Sizing]] | Measurement-first sizing for llama.cpp shapes: slot pools against a headroom band, per-device accounting, prompt cache. |
| [[LLM Performance Diagnosis]] | Probing why a served model is slow: the flat-depth probe, offload duty-cycle arithmetic, and step-shape attribution. |
| [[Model Swap and Rollback]] | The model-swap discipline: fit preflight, fetch verification, build and flag checks, live validation, three-pin revert. |
| [[PCIe Link and Atomics]] | Link probing, the streaming-versus-compute ceiling arithmetic, and the atomics wall that decides multi-GPU collectives. |
| [[Quantization by Hardware Architecture]] | This page maps LLM quantization formats to the GPU and CPU architectures that support them. |
| [[Quantization Production]] | — |
| [[qwen3.8-strix-optimizations]] | This page answers one question: **how do we make Qwen3.8-27B serve faster on |
| [[RAM CPU Offloading in VLLM]] | vLLM uses CPU RAM and swap space to extend GPU memory capacity. |
| [[RDNA4 Container Serving]] | Bringing a model larger than VRAM onto Dankserver's 2x R9700 (gfx1201) with the tcclaviger vLLM stack — quant-class… |
| [[RDNA4 MoE Expert Offload VRAM Budget]] | The expert-offload plan arithmetic on the tcclaviger fork — which terms a deploy can move, the util law, the… |
| [[Running Qwen3.6 27B FP8 on Dankclaw]] | Qwen3.6-27B is a dense vision-language model from the Qwen3.6 series. |
| [[Running Qwen3.6 35B A3B FP8 on Dankserver]] | Qwen3.6-35B-A3B is a Mixture-of-Experts vision-language model from the Qwen3.6 series. |
| [[Quantization Production]] | Making a quantized payload here: the conversion recipe, the scoring gate, and the campaign ladder behind a shipped file. |
| [[RDNA4 MoE Expert Offload VRAM Budget]] | Expert-offload arithmetic on the RDNA4 fork: movable terms, the utilisation law, graph charge, ceilings, context ladder. |
| [[Shared Memory Architecture Dankclaw]] | Dankclaw uses AMD Strix Halo APU with unified memory architecture. |
| [[swift15-campaign-worklog]] | — |
| [[Tcclaviger Fork Ops]] | Operating the tcclaviger RDNA4 vLLM fork on dankserver (shape G era, rollback target since the 2026-10-05 FreeToken… |
| [[ubatch-benchmark-large]] | Standardized ubatch sweep for the active kingjones Qwen3.8 ROCmFPX deployment. |
| [[vLLM Boot Forensics and Config Interrogation]] | Capturing a vLLM container boot failure's real root cause from journald (the container log is already gone),… |
| [[VLLM Index]] | This index contains all documentation for vLLM deployment on Dankserver and Dankclaw. |
| [[VLLM Overview]] | vLLM is a fast and easy library for large language model inference and serving. |
| [[vLLM Porting and Compile Triage]] | Forward-porting a model architecture to a new vLLM major (additive-only discipline, gap enumeration, shims that… |
| [[vllm_research_findings]] | vLLM v0.26.0 (latest stable, released July 25, 2026) supports DFlash speculative decoding with the `--spec-method… |
| [[x4-pair-experiment]] | What mainline llama.cpp can and cannot do with a 3090 + Strix Halo combination, measured 2026-09-16 on Dankclaw — the… |
## Pages
| Page | Owns |
|---|---|
serverAppConfigs-docs/LLM/kingjones-q8agent-rocmfpx.md
-241
View File
@@ -1,241 +0,0 @@
---
type: reference
description: "**Status (historical, 2026-09-13):** was to deploy on Dankclaw."
tags:
- llm
- rocm
- frank
timestamp: 2026-09-15
---
# frank-dragoon: kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT
> **SUPERSEDED 2026-09-15.** The Q8_0_ROCMFPX-AGENT checkpoint (with the
> separate `draft-mtp` head) now serves Dankserver as **frank-medium** on the
> fork's fixed build — see [[frank-medium-rocmfpx]]. The failure below was
> specific to the gfx1151 legacy-read path and the Q6_0 single-file era.
**Status (historical, 2026-09-13):** was to deploy on Dankclaw. Serving is
**BLOCKED by
the `set_rows` build bug** — the model loads but crashes in
`ggml_compute_forward_set_rows` before it can serve. All config goals are met;
a build fix is required for the model to run.
**Alias:** `frank-dragoon`, port **7770**.
The large service serves the kingjones `Qwen3.8-27B-ROCmFPX-Q8_0-AGENT` GGUF
from `/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT/` with a
charlie12345/ROCmFPX `llama-server` (build `build-kairic`). It runs a
**standalone MTP draft head** (loaded separately, on ROCm0), **TurboQuant** V
KV cache, the **Sharp** chat template with **reasoning on**, and a **full
262144 context / 1 slot** profile. The whole payload plus the 262k KV cache
fit in the lifted **114 GiB TTM/GTT** budget (see the 114 GiB change in
[[Dankclaw Improvements]] and [[qwen3.8-strix-optimizations]]).
| Figure | Value |
| ------ | ----- |
| Model | kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT (`Q8_0_ROCMFPX`) |
| HF repo | `kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF` (public) |
| Runtime | charlie12345/ROCmFPX `llama-server`, `build-kairic` |
| Draft head | standalone `mtp-Qwen3.8-27B-Q4_0.gguf` on ROCm0 |
| KV cache | f16 K, TurboQuant (`turbo4`) V, full 262144 context, 1 slot |
| Reasoning | on, `deepseek`, budget `-1`, `--reasoning-format none` |
| Chat template | Sharp (repo-tracked), `enable_thinking` + `preserve_thinking` on |
| Host path | `/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT/` |
| Tunables | `scripts/frank-dragoon.env` |
| Serve script | `scripts/serve-frank-dragoon.sh` |
## Serving status: the `set_rows` build bug (blocks serving)
As of 2026-08-25 the Q8_0 AGENT model will not serve on Dankclaw. It loads,
reaches `initializing slots, n_slots = 1`, but never prints `start serving`.
Health stays `503` until the process exits. The crash is in
`ggml_compute_forward_set_rows` in `libggml-cpu.so.0`, a NULL function-pointer
jump (address `0x00000000`).
The backtrace (gdb). The main thread and one OpenMP worker crash in the same
path:
```mermaid
graph TD
A[llama_context::graph_compute] --> B[ggml_backend_cpu_graph_compute]
B --> C[ggml_graph_compute]
C --> D[GOMP_parallel]
D --> E[ggml_compute_forward_set_rows]
E --> F[NULL dispatch -> address 0x0]
D --> G[Thread 23 worker: same path]
```
Test matrix. Five ROCmFPX builds and the all-CPU path were tested on a scratch
port (`/tmp/g80bench/run.sh`), `gl=45`, `ctx=131072`. All crash in `set_rows`.
| Build | Binary | Version | Result |
| ----- | ------ | ------- | ------ |
| build-kairic | `ciru-qwen38/ROCmFPX/build-kairic/bin/llama-server` | v81 (205a3e5f4) | segfault in set_rows |
| build-promptforge | `ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` | PromptForge/ActiveFPX lineage | segfault in set_rows |
| build-rocmi4-w4a4 | `rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server` | v257 (c49ebdb) | segfault in set_rows |
| build-strix-rocmfp4 | `GitPackages/ROCmFPX/build-strix-rocmfp4/bin/llama-server` | v209 (00d5452) | segfault in set_rows |
| all-CPU | build-kairic, `-ngl 0` | v81 | segfault in set_rows |
The crash is not GPU-specific (the all-CPU path fails too), not MTP-specific
(`--spec-type none` fails), and not fixed by `LLAMA_SET_ROWS=0`. So it is a
build bug: the Q8_0 AGENT GGUF graph needs a `set_rows` dispatch the ROCmFPX
backends do not register. The older caf-eemin ROCmI4 and Kairic Edge models
served fine on related ROCmFPX builds. The caf-eemin payload was about 14.4
GiB; the separate
PromptForge/ActiveFPX deployment used a much larger multi-file payload of about
36.7 GiB. These are different deployments. The Q8_0 AGENT graph is the trigger
for the `set_rows` failure.
**Fix path:** rebuild charlie12345/ROCmFPX from source with a working
`set_rows` for the ROCmFPX backend, or find a build where it is registered.
This unblocks the model. See the context budget reality below once `set_rows`
is fixed.
## Model files
Three files, from the HF repo above:
| File | Size | Purpose |
| ---- | ---- | ------- |
| `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` | 28.21 GB | main weights |
| `mtp-Qwen3.8-27B-Q4_0.gguf` | 1.68 GB | standalone MTP draft head (loaded by the script) |
| `mmproj-Qwen3.8-27B-BF16.gguf` | 0.93 GB | vision projector (`--mmproj`) |
The repo also has a `mtp-Qwen3.8-27B-Q8_0.gguf` (3.16 GB) draft; the env uses
the Q4_0 draft. Download them from the resolve URLs below (they are large —
use the token in `~/.cache/huggingface/token`).
## Download
```bash
# on Dankclaw
TOKEN=$(cat ~/.cache/huggingface/token)
REPO=kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF
DIR=/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT
mkdir -p "$DIR"; cd "$DIR"
dl(){ curl -fsSL -C - -o "$1" --retry 5 --retry-delay 3 --retry-max-time 3600 \
-H "Authorization: Bearer $TOKEN" \
"https://huggingface.co/$REPO/resolve/main/$1"; echo "done: $1"; }
dl Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf
dl mmproj-Qwen3.8-27B-BF16.gguf
dl mtp-Qwen3.8-27B-Q4_0.gguf
```
The serve script checks each artifact is present and readable before launch
(`scripts/frank-dragoon.env` -> the four `MODEL_*` files plus the chat
template). A missing file fails the artifact check with `missing or
unreadable required artifact`.
## Serving profile
`scripts/serve-frank-dragoon.sh` builds the `llama-server` call from the env:
```
/dev/llama-server -m <main> --alias frank-dragoon
--host 0.0.0.0 --port 7770 --jinja
-dev ROCm0 -ngl 999 -np 1 -c 262144
-b 512 -ub 512 -fa on
-ctk f16 -ctv turbo4 -t 16 -tb 32
-ctxcp 32 --cache-ram 8192 --cache-prompt --cache-idle-slots --metrics
--spec-type draft-mtp --model-draft <mtp-Q4_0>
--mmproj <mmproj-BF16>
--spec-draft-device ROCm0 --spec-draft-ngl 999
--spec-draft-type-k f16 --spec-draft-type-v turbo4
--spec-draft-n-max 4 --spec-draft-n-min 0
--spec-draft-p-min 0.0 --spec-draft-p-split 0.10
--spec-draft-backend-sampling
--temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0
--presence-penalty 1.5
--reasoning on --reasoning-format none --reasoning-budget -1
```
Key choices:
- **Standalone MTP draft** (`--model-draft`, `--spec-type draft-mtp`). The
AGENT GGUF has no embedded draft head; the draft head is a separate file on
ROCm0. This matches the Qwen3.8-27B AGENT shape (`--spec-draft-type-k
f16 --spec-draft-type-v turbo4`).
- **f16 K + TurboQuant V** cache. f16 K is the safe long-ctx default; TurboQuant
halves the V cache at 262k while keeping quality (see the ubatch/threads-batch
notes in [[qwen3.8-strix-optimizations]]).
- **262144 context, 1 slot — the target.** The whole context KV cache plus the
~30 GB payload (main + draft + mmproj) are meant to fit in the 114 GiB
budget. Measured reality: `gl=999` (all layers on ROCm0) at 262144 **OOMs**
on device-local (`cudaMalloc failed`, the KV cache needs ~10-18 GiB that does
not fit once the payload is on the card). `gl=45` (partial offload) fits a
131072 context. The env ships `CTX_SIZE=262144`; the proven serving context
today is **131072 at `gl=45`** (before the `set_rows` fix). `deploy.sh` waits
up to 20 minutes (see `LLM_WAIT`).
- **Reasoning on.** The Sharp template drives thinking (medium effort);
`--reasoning-format none` lets the template project the reasoning content into
`reasoning_content`. `--reasoning-budget -1` = no budget cap.
## Reasoning + chat template
The Sharp template (`scripts/chat-template.jinja`, repo-tracked) is a
Qwen-AGENT shape. Env `CHAT_TEMPLATE_KWARGS` keeps the reasoning trail in the
conversation:
```json
{"enable_thinking": true, "preserve_thinking": true, "preserve_reasoning": true}
```
`preserve_thinking: true` keeps the model's prior thinking in the next prompt
(the user's "preserve thinking" request). `preserve_reasoning: true` keeps the
model's prior reasoning trace across requests too (OpenWebUI sends
`preserve_reasoning`/`preserve_thinking: true` when the conversation needs the
full trail). For multi-turn reasoning, the template stays in the reasoning
shape and `enable_thinking: true` per request.
## Memory: the 114 GiB TTM/GTT requirement
The 262k f16 KV cache is large (~a few GB) and the payload is ~30 GB. Both live
in the Strix Halo unified-memory GTT. The TTM `pages_limit` was raised to
114 GiB on 2026-08-25 (written live to
`/sys/module/ttm/parameters/pages_limit`, then persisted in
`/etc/modprobe.d/ttm.conf`). Without the raise the effective ceiling is ~15 GiB
and a 28 GB + 262k-KV load stalls. The GTT page-limit change is per the
`qwen3.8-strix-optimizations` table item 5.
## Health + troubleshooting
- Health: `curl http://192.168.0.143:7770/health` → `{"status":"ok"}`.
- `/api/models` reports `n_ctx:262144, n_ctx_train:262144, n_params:27.3B`.
- Model load is slow (a few minutes cold for the ~30 GB payload with a fresh
KV cache). Health returns 503 while loading. Do not kill the process.
`deploy.sh` waits up to 20 minutes (see `LLM_WAIT`).
- Kernel spam about `amdgpu: SVM mapping failed` is the known documented
symptom, not root cause (see the crash-loop procedure in the project
AGENTS.md). Investigate other causes before assuming memory pressure.
- A client abort logs `W srv stop: cancel task, id_task = N`; a natural stop
logs only `slot release` with `truncated = 0`.
## Env tunables
`scripts/frank-dragoon.env`. Serving: `CTX_SIZE=262144`,
`PARALLEL=1`, `BATCH_SIZE=512`, `UBATCH_SIZE=512`, `NUM_THREADS=16`,
`THREAD_BATCH=32`, `CACHE_TYPE_K=f16`, `CACHE_TYPE_V=turbo4`,
`CTX_CHECKPOINTS=32`, `CACHE_RAM=8192`. Reasoning: `REASONING=on`,
`REASONING_FORMAT=none`. Sampling: `TEMPERATURE=0.7`, `TOP_K=20`,
`TOP_P=0.8`. MTP: `MTP_N_MAX=4`, `MTP_N_MIN=0`, `MTP_P_MIN=0.0`,
`MTP_P_SPLIT=0.10`. Strix Halo: `HSA_OVERRIDE_GFX_VERSION=11.5.1`,
`GGML_HIP_ENABLE_UNIFIED_MEMORY=1`.
**Never name a tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`** — the ROCm HIP
runtime reads those bare vars (see the 2026-08-13 TEMP incident). llama.cpp
reads `--host/--port/--temp` as CLI flags, never as env vars.
## History
- **2026-08-25 (today):** switched the large service to the kingjones
Q8_0-AGENT model with 262144 context and reasoning on. Downloaded the three
payload files from the HF repo, raised the TTM/GTT to 114 GiB for the full
context, and persisted the env/serve script in this repo. Measured that the
model loads but crashes in `ggml_compute_forward_set_rows` on all five
available ROCmFPX builds; serving is BLOCKED pending a rebuild. The config
goals (GTT 114 GiB, reasoning on, preserve_thinking/preserve_reasoning on,
262144 env context) are in place. Proven serving context today is 131072 at
`gl=45`. See the `set_rows` build bug section above.
- Preceding model: Kairic Edge Qwen3.8-27B IU4 (deployed 2026-08-25 ~05:42,
retired the same day for the AGENT model). See [[frank-dragoon]].
serverAppConfigs-docs/LLM/ling-3.0-flash-trial.md
-116
View File
@@ -1,116 +0,0 @@
---
type: reference
description: "`bailingmoe3-support` (commit `db4480bc8`, build b10411) at `/home/dank/llama.cpp-ling`."
tags:
- llm
timestamp: 2026-09-03
---
# Ling-3.0-flash Q6_K Trial on Dankclaw
> **RETIRED 2026-08-17.** The `frank-dragoon-ling` unit is disabled under the boot policy. At this page's retirement date `frank-dragoon` ran the Qwen3.8-27B PP head (since superseded by the STRIX deployment — see [[frank-dragoon]]), not the Qwen3.6-27B config described in the (obsolete) switch-back section below.
## Status
- Service: `frank-dragoon-ling` (systemd).
- Model: `inclusionAI/Ling-3.0-flash` Q6_K (MoE, ~118B total params).
- Runtime: llama.cpp **fork** `aetherbird/llama.cpp` branch
`bailingmoe3-support` (commit `db4480bc8`, build b10411) at
`/home/dank/llama.cpp-ling`. Stock llama.cpp cannot load the
`bailingmoe3` architecture.
- Backend: `Vulkan0` (RADV) with AMD GTT.
- Spec: `draft-mtp` `n_max 3`. The MTP block is bundled in the Ling GGUF
itself (block 42) - no separate drafter file.
- Context: 262144 total, 2 slots x 131072, batch 512 / ubatch 256.
- Measured: load ~45s, generation ~7.7 t/s (cold, 4k prompt) to ~36 t/s
(tiny cached prompt). Concurrent streams both complete.
## Why Vulkan + GTT
The Strix Halo APU has 8 x 16 GB LPDDR5 modules = **128 GB physical
shared memory**. The firmware carves ~96 GiB as dedicated iGPU VRAM; the
kernel sees the remainder (~30 GiB, `MemTotal`). GRUB adds no memory
options (`iommu=pt` only).
The ROCm runtime reports 96 GiB VRAM. **Vulkan (RADV) reports a
111.2 GiB heap** (113910 MiB) because it can also reach system memory
through the GTT. For a model near 100 GiB, Vulkan is the only backend
with room. `--list-devices` shows the heap:
```
ROCm0: AMD Radeon Graphics (98304 MiB)
Vulkan0: AMD Radeon Graphics (RADV STRIX_HALO) (113910 MiB)
```
## Memory Fit (Q6_K = 98.26 GiB)
| Item | Size |
| ---- | ---- |
| Vulkan heap (llama.cpp view) | ~111.2 GiB |
| Model Q6_K file | 98.26 GiB |
| Model + MTP + graphs in heap | ~98.7 GiB |
| Free heap after load | ~10 GiB |
Ling uses gated MLA and KDA attention. KV cache stays tiny: 256k total
context costs only ~2.5 GiB. This is why 256k context fits.
## Freeze Bug (2026-08-14) - Spec Path Deadlock
### Symptom
Under 2+ concurrent streams with speculative decoding, one stream hard-freezes
mid-generation (zero output for minutes; a deadlock, not slowness). Freeze
point varies (385-1588 tokens). Single streams never freeze.
### Isolated cause
- Not context exhaustion: `truncated = 0`, under 131072 ctx.
- Not batch (3072 and 8192 both froze) and not log verbosity (0 and 4 both).
- `parallel=2` + `draft-mtp,ngram-mod` -> freeze. SPEC OFF -> no freeze.
- Culprit: the speculative-decode / context-checkpoint path in llama-server
with multiple slots on the Vulkan backend. Known bug family: PR #24891,
#24797 (server checkpoint fixes), #25144 (MTP draft crash), plus a
ggml-vulkan hang-under-sustained-compute class.
### Resolution: SPEC OFF
`SPEC_TYPE=none`, `PARALLEL=2`. All concurrent streams complete (EOS).
Speed cost: ~10-13 t/s under contention vs ~28-32 t/s single-stream with
MTP. Fallback chain if spec-off regresses: A) `PARALLEL=1` keep MTP,
B) switch back to Qwen3.6-27B (frank-dragoon, MTP n_max 3).
## Files
- `scripts/frank-dragoon-ling.service` - systemd unit.
- `scripts/serve-ling-flash.sh` - serve script (fork binary, Vulkan,
draft-mtp).
- `scripts/frank-dragoon.ling.env` - tunables.
Passwordless management is installed:
`/usr/local/sbin/frank-restart-ling.sh` and `/etc/sudoers.d/frank-ling`.
## Switch Back (obsolete)
The switch-back commands below are obsolete. At retirement time
`frank-dragoon` ran the Qwen3.8-27B pipeline-parallel head. To restore a Ling trial,
re-enable `frank-dragoon-ling` from the repo unit
(`scripts/frank-dragoon-ling.service`) and serve script
(`scripts/serve-ling-flash.sh`), then start it on a non-conflicting port.
## Caveats
- The `bailingmoe3` fork validates CPU and CUDA upstream. Vulkan was
validated here (load + generation + concurrent streams) on 2026-08-14.
- The tokenizer logs `special_eos_id is not in special_eog_ids` at load.
It is a warning; generation stops correctly at EOS.
- Physical RAM pressure stays low because the model fits in the UMA
carve-out. Generation speed on this APU is below the Qwen3.6-27B MTP
config (~7.7 vs ~15-19 t/s at 4k).
- The heap is effectively full (98.7 of 111.2 GiB). Do not raise the
context much without checking `--list-devices` free MiB.
## See Also
- [[frank-dragoon]]
- [[DFlash DSpark Speculative Decoding]]
- [[deployment]]
serverAppConfigs-docs/LLM/ling-3.0-flash-vllm-plan.md
-22
View File
@@ -1,22 +0,0 @@
---
type: guide
description: "This page was the plan for a vLLM-ROCm deployment of `jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native` (INT4 W4A16, ~62 GB) on"
tags:
- llm
- vllm
- rocm
timestamp: 2026-09-03
---
# Ling-3.0-flash on Dankclaw — vLLM-ROCm Deployment Plan (never shipped)
This page was the plan for a vLLM-ROCm deployment of
`jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native` (INT4 W4A16, ~62 GB) on
Dankclaw. The deployment never shipped. The actual Ling work used the
llama.cpp Q6_K fork trial instead: [[ling-3.0-flash-trial]].
The trial service (`frank-dragoon-ling`) was retired and disabled under the boot
policy on 2026-08-17, in the same redo that renamed `frank-dragoon` to the
Qwen3.8-27B PP head ([[frank-redo-2026-08-17]]). Two fleet generations later the
name belongs to the halogen flash-server endpoint ([[frank-dragoon]]). See git
history for the full decision table.
serverAppConfigs-docs/LLM/qwen3.8-spec-matrix.md
-141
View File
@@ -1,141 +0,0 @@
---
type: reference
description: "This page records the speculative-decoding benchmark history and the"
tags:
- llm
- speculative
timestamp: 2026-09-03
---
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
> See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
> [deployment.md](deployment.md) for the active configuration.
# Qwen3.8-27B Speculative-Decoding Matrix
This page records the speculative-decoding benchmark history and the
production tuning for the then-Qwen3.8-27B service (`frank-dragoon`), at the
time a pipeline-parallel head with a Dankserver worker.
## Production state (2026-08-17)
The pipeline-parallel shape this matrix was tuned for — Q6_K target, standalone
Q8_0 draft on ROCm0 only, `--tensor-split 0.68,0.32` (45 of 65 blocks), K `q8_0`
/ V `f16`, pool 262144 at one slot, worker steady at 23854/24576 MiB — is
recorded once, in [[Pipeline Parallelism/Qwen3.8 PP Deployment]] §Production
state, together with the split-limit arithmetic (why 45 blocks is the maximum),
the KV-per-layer bill and the FA-all-quants rebuild the mismatch needed. That
era's endpoint identity is an [[frank-dragoon]] era. This page keeps only the
speculative-decoding measurements.
## How MTP works on Qwen3.8 (bundled-head era)
The unsloth `UD-*` GGUFs keep the MTP (Multi-Token Prediction) head inside
the model file. llama.cpp b10411+ auto-detects it and creates the draft
context from the target model. Do not pass `--spec-draft-model` for those
files. This is superseded: the production target of 2026-08-17 carried no
bundled head, and the serve script passed the standalone draft sidecar with
`--spec-draft-model` (shape: [[Pipeline Parallelism/Qwen3.8 PP Deployment]]).
The stock llama.cpp build must be b10411 or newer. The 2026-08-14 rebuild
on Dankclaw is b10433 (`9b05354ec`) with `GGML_HIP` + `gfx1151` + Vulkan.
## History (superseded configurations, 2026-08-14 to 08-16)
All superseded configs used `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB,
bundled MTP) before the Q6_K swap. The ngram-mod preset survives as the
fallback layer under MTP. The Vulkan0 verdict from this era applies to the
standalone configuration only; the PP head runs ROCm0.
- **2026-08-14: bundled MTP** — `draft-mtp`, n_max 5, parallel 2, ctx
524288 (2 x 262144). ~13.3 t/s single, ~8 t/s per slot under 2
concurrent streams. No premature stop.
- **2026-08-15: DSpark sidecar** — `magnitudedev/Qwen3.8-27B-DSpark-GGUF`
(Q8_0, 1.36B, block_size 7), n_max 7, PARALLEL=1. Two concurrent 16k
prose prompts crashed the ROCm backend with `ROCm error: an illegal
memory access` in `common_speculative_impl_draft_dflash::draft` (status
6/ABRT, systemd restart, ~503 during reload). DSpark reuses the DFlash
block-diffusion machinery, which shares one draft context across
parallel slots; concurrent block drafting faults in HIP stream sync.
This is the same multi-slot bug class that retired `draft-dflash` (premature
EOS, retired 2026-08-14). The crash shows on 16k concurrent streams, not on a
67k pair — concurrency is the trigger, not size. Fix: PARALLEL=1 (requests
queue safely). Retired the same day.
- **2026-08-15 to 08-16: `--spec-default` (ngram-mod), Vulkan0** — n_match
24 / n_min 48 / n_max 64, draftless, PARALLEL=2 (2 x 32k parallel stream
test passes). Chosen for parallel safety after DSpark; kept as the
fallback under MTP in the 2026-08-17 shape.
## Benchmark Method
The matrix ran 54 configurations: 3 spec types (`draft-mtp,ngram-mod`,
`draft-mtp`, `ngram-mod`) x `--spec-draft-n-max` 2..10 x
`--spec-draft-p-min` {off, 0.8}. Each config loaded `llama-server`,
then ran 3 generations. The tool is `scripts/spec-matrix-bench.sh`;
analysis is `scripts/analyze-spec-matrix.py`.
## Matrix Caveat: Benchmark Prompt
The matrix used a repetitive 1100-token prompt
(`scripts/bench_live.sh` style). MTP drafts that repeated sentence almost
perfectly, which inflated the numbers for long drafts. The matrix reported
`draft-mtp` n_max=10 at ~30 t/s. A realistic story prompt gave very
different results:
| n_max | median t/s (realistic) |
| ----- | ----------------------- |
| 4 | 12.5 |
| 5 | 13.3 (peak) |
| 6 | 10.6 |
| 7 | 10.1 |
| 10 | 8.6 |
| none (baseline) | 7.2 |
Long drafts collapse to ~18% acceptance on real writing. n_max=5 balances
acceptance against draft cost.
## Premature Stop Investigation
Two failure patterns appeared during early production tests:
1. Speed collapse. `ubatch >= 512` reduced MTP acceptance to ~25% and
generation to ~9 t/s. `ubatch 128` with `batch >= 2048` triggered a
premature instant-EOS under 2 concurrent slots. `ubatch 256` avoids
both: ~30 t/s single, concurrent-safe.
2. Apparent premature-EOS. The repetitive benchmark prompt with `temp 1.0`
sometimes stops after 1-2 tokens because the model legitimately emits an
EOS on a wall of repeated text. This is sampling variance, not a
corruption bug. Realistic prompts never stop early.
## High-Context Backend Comparison (2026-08-15)
Tool: `scripts/bench-high-ctx.sh <device> <ctx>`. Both runs at ctx 262144
with a ~200k-token realistic-prose prompt, `--spec-default`, batch 8192 /
ubatch 256, single slot, on scratch port 7892.
| Device | Prompt tokens | Prefill (t/s) | Generation (t/s) | Wall (single req) |
| ------ | ------------- | ------------- | ---------------- | ----------------- |
| ROCm0 | 202822 | **119.1** | 3.83 | **1770 s** |
| Vulkan0 | 200235 | 106.3 | **5.68** | 1929 s |
Verdict: at very high context, Vulkan0 decodes **48% faster** (5.68 vs
3.83 t/s); ROCm0 prefills 12% faster. Generation is the interactive cost
(users stream tokens). This verdict applies to the standalone configuration; the PP
head runs ROCm0 — HIP registers before Vulkan and the iGPU dedup rule (the llama.cpp
PR #23897 workaround) keeps the first backend's device.
Both `tg_3s` values were rock-steady across the run (3.83 and 5.67 t/s).
High-context decode collapses on both backends versus ~25 t/s at 16k ctx;
that is the long-context attention penalty, not a spec artifact.
## Where the tunables live
That era's head and worker env/units are listed (and marked deleted) in
[[Pipeline Parallelism/Qwen3.8 PP Deployment]] §Files. `scripts/frank-dragoon.env`
is no longer a PP-head env at all — it is the halogen-flash-server trio of the
current era, which [[frank-dragoon]] owns.
## See Also
- [[frank-dragoon]]
- [[ling-3.0-flash-trial]]
- [[deployment]]
serverAppConfigs-docs/LLM/qwen3.8-strix-optimizations.md
-232
View File
@@ -1,232 +0,0 @@
---
type: reference
description: "This page answers one question: **how do we make Qwen3.8-27B serve faster on"
tags:
- llm
timestamp: 2026-09-03
---
# Qwen3.8-27B on Strix Halo: Speed Without Lowering the Quant
This page answers one question: **how do we make Qwen3.8-27B serve faster on
Dankclaw (Strix Halo, Radeon 8060S) while keeping UD-Q8_K_XL?** It records
the model's real architecture, the levers that are available, their evidence,
and the test matrix planned for 2026-08-16 morning.
Related pages:
- [Speculative-decoding matrix](qwen3.8-spec-matrix.md)
- [[frank-dragoon]] — the endpoint this host became; its Eras and Laws carry the
backend and spec-decode measurements.
- [Dankclaw Improvements](Dankclaw%20Improvements.md) (OS-level items)
- [Pipeline Parallelism: Qwen3.8 PP Deployment](Pipeline%20Parallelism/Qwen3.8%20PP%20Deployment.md)
## TL;DR — ranked levers
| # | Lever | Expected gain | Cost | Status |
|---|-------|--------------|------|--------|
| 1 | Pipeline-parallel to the RTX 3090 (RPC) | ~1.5–2× decode | 2.5 GbE link, both services stop for test | Kit ready, test 08-16 |
| 2 | MTP `n_max` explicit 4–5 + Q4_0 draft head | +5–15% on the spec share | restart, 1.56 GB file | test 08-16 |
| 3 | Plain `Q8_0` A/B vs UD-Q8_K_XL (same 8 bits) | dequant ALUs may differ | restart, 29 GB file | test 08-16 |
| 4 | Prefill path: ROCm backend + tuned ubatch/threads-batch | 2–3× prefill for agentic turns | per-workload backend choice | partial data |
| 5 | TTM `pages_limit` raised | removes the hard VRAM ceiling; the full 262k f16 KV plus the Q8 AGENT payload, MTP draft, and mmproj fit in one slot | modprobe, set live 2026-08-25 | applied — value and file: [[Dankclaw System Specs]] (Memory) |
| 6 | Vulkan ICD pinning + loader-layer disable | robustness; small | service env vars | test 08-16 |
Everything else (quantization downgrade, NVFP4, larger drafts) is documented
below for completeness but is out of scope per the "no lowering" constraint.
## What the model actually is
Verified from the GGUF header (`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL.gguf`,
dumped 2026-08-15) and the official model card.
- Architecture: `qwen35` / Qwen3.8-27B, **hybrid attention**.
- 64 main layers + 1 MTP (nextn) layer = 65 GGUF blocks.
- 48 Gated DeltaNet (linear/recurrent) layers + 16 gated full-attention
layers, in a 3:1 pattern (`full_attention_interval = 4`).
- Hidden 5120, FFN 17408, 24 Q heads / 4 KV heads, head dim 256.
- Native context 262144 (extendable per the card; production runs 262k/slot).
- GDN layers keep a **fixed-size** recurrent state (constant per layer, not
per token). Only the 16 full-attention layers grow with context.
- KV per token (full-attention part only): 16 layers × 2 (K,V) × 4 heads ×
256 dim × 1 byte (q8_0) = **32 KiB/token**. At 65536 ctx ≈ 2 GiB; at
262144 ctx ≈ 8 GiB.
- Sampling defaults (card, thinking mode on by default): temperature 1.0,
top-k 20, top-p 0.95, min-p 0.0. Production already matches.
Consequences of the hybrid design:
1. **Decode is cheap in memory.** Attention cost scales with only 16 layers;
the 48 GDN layers are O(1) per token. Long-context decode degrades far
slower than a dense transformer.
2. **Prompt cache reuse is unavailable upstream.** Recurrent/SSM state
cannot be prefix-cached the way attention KV can. Every agentic turn
re-prefills the whole conversation. **Prefill is the hidden cost of
agentic use**, and it is the dominant cost we have not yet optimized.
3. **MTP is native and strong.** The model ships a nextn head; on Vulkan at 16k
it roughly doubled the no-spec rate (the pair of figures is
[[qwen3.8-spec-matrix]]'s realistic-prompt table).
## Baseline this page optimized (2026-08-15)
`frank-dragoon` then ran llama.cpp on the iGPU with the UD-Q8_K_XL target
(bundled MTP head), `--spec-default` ngram-mod under a draft-mtp first link, two
262144 slots, q8_0 KV, batch 8192 / ubatch 256, threads 12. That era's ledger is
[[frank-dragoon]]'s; what this page keeps is the shape of the problem: the
service occupied **28 GiB of the then 96 GiB firmware VRAM carve, ~74 GiB
idle**, decode held ~25 t/s at 16k and collapsed at 262k depth
([[qwen3.8-spec-matrix]] §High-Context Backend Comparison), and prefill was the
unoptimized agentic cost (section 4).
## The levers, in detail
### 1. Pipeline-parallel to the RTX 3090 (biggest win)
The 3090 on Dankserver had headroom that night (it carried the day's small-model
service with a couple of GiB free — that card's eras are [[frank-zealot]]'s).
llama.cpp RPC splits the model by **layers**: each device runs its layer range,
hidden states cross the 2.5 GbE ethernet once per token. The per-position wire
bill is measured in [[Pipeline Parallelism/Ethernet Bandwidth Math]] §0; at
~0.5 ms LAN RTT the bandwidth cost is negligible — the split is latency-bound,
not bandwidth-bound.
Why this is a real speedup and not just "two GPUs": the 3090 does
matmuls roughly 2–3× faster per layer than the 8060S (936 vs 512 GB/s
memory, 80 vs 40 CUs). A 50/50 layer split balances per-layer latency, so
each token step takes ≈ max(half-on-3090, half-on-Strix) + one RTT instead
of all-64-layers-on-Strix.
Every decision this led to is written once, in
[[Pipeline Parallelism/Qwen3.8 PP Deployment]] §Gotchas (with the 4B smoke test
that proved each): head on the box that already holds the model file, **no
`--device` on the head** (setting it silently drops RPC), the fixed
HIP-before-Vulkan device selection, the reported-free-memory split trap, the
identical-build requirement, and `-dio` on the head — the last one visorcraft's
Strix finding, matching our own 28 GiB load-stall history. Scripts and the
morning runbook are on the same page.
### 2. Speculative-decoding tuning (same bits, free tokens)
- **MTP `n_max`.** Production left `--spec-draft-n-max` unset; the fleet matrix
(Vulkan, 16k, realistic prompt) found the depth-5 peak against the no-spec
baseline — both figures are [[qwen3.8-spec-matrix]]'s realistic-prompt table.
The published Strix tuning curve peaks at n_max 3–4 (30.3 t/s at 4 vs 28.9 at
3 vs 29.9 at 5). The PP env set **4** explicitly; the matrix re-tested 3/4/5
on the real split.
- **Q4_0 draft head.** The published Strix card measured a standalone
Q4_0 MTP head ~6–10% faster than Q8_0 for the same acceptance (the draft
head is a 248k-vocab output projection; cheaper dequant, same accuracy).
The bundled head inside UD-Q8_K_XL is Q8-class; the standalone
`ggml-org/mtp-Qwen3.8-27B-Q4_0.gguf` (1.56 GB) is being downloaded for
an A/B via `--spec-draft-model`.
- **Ngram-mod fallback stays.** It is the only spec type that survived parallel
load; the DSpark ROCm fault and the DFlash multi-slot corruption are timed and
explained on [[qwen3.8-spec-matrix]].
### 3. Same-bit quant A/B: UD-Q8_K_XL vs plain Q8_0
Both are 8-bit. UD (unsloth "dynamic") mixes block sizes per tensor; plain
`Q8_0` uses uniform 32-element blocks. Dequant of a smaller block costs
more ALUs per element; on a memory-bound GPU the difference is usually
small, but Strix's iGPU has tight ALU headroom. `ggml-org/Qwen3.8-27B-GGUF`
ships a Q8_0 (29.12 GB). One restart to test; if it wins, same quality at
zero quality cost.
### 4. Prefill is the agentic bottleneck — route it right
Every agentic turn re-prefills the whole conversation (no prefix caching for
hybrid state, upstream). Levers:
- **Backend choice per workload.** Measured @64k on this box (b10433, the era
[[frank-dragoon]] owns): ROCm 410 t/s vs Vulkan 302 t/s prefill; Vulkan 25.1
vs ROCm 24.3 t/s decode. For chat-heavy use: Vulkan. For agentic/batch-heavy
use: ROCm. The PP head (ROCm side) gets this for free.
- **ubatch/threads-batch.** Production runs ubatch 256. Larger ubatches
(512/1024) fill the iGPU better during prefill; `--threads-batch` for
the CPU-bound tail. Untested at prefill scale on this box — matrix item.
- **KV q4_0** for very long contexts: halves the 16-layer KV (8 GiB → 4 GiB
at 262k). Only worth it if long-ctx attention is the bottleneck; q8_0 is
the safe default.
### 5. OS level (from the Improvements doc, re-verified 2026-08-15)
- **TTM `pages_limit`** — the raise itself (value, live write, modprobe file,
date) is one row on [[Dankclaw System Specs]] (Memory). What matters to this
page is the mechanism: the driver-reported 96 GiB VRAM carve-out (VGM/SVM
pool) was never the ceiling — the TTM kernel page map was, and until it was
raised a 262k f16 KV cache plus a ~28 GB payload could not load in one slot.
- **Boot line** — IOMMU policy on this host is [[dankclaw-iommu-translation-off]]'s
and [[Dankclaw System Specs]]'s; it has changed twice since this lever was
written, so no era-scoped reading of `/proc/cmdline` belongs here.
- **Vulkan ICD pinning** — `radeon_icd.json` exists; no `VK_ICD_FILENAMES`
or loader-layer env is set. Pinning the radeon ICD and disabling loader
layers (`VK_LOADER_LAYERS_DISABLE=all`) is the recommended Strix
hardening; testable as service env vars, zero risk.
- **Power** — GPU DPM is already `performance`, forced level `auto`. The
`power_profile` sysfs file assumed in the Improvements doc does not exist
on this machine (checked 2026-08-15); that item is N/A here.
- Threads: 12 of 16 cores in production. Community reports 12–16 are
equivalent; leave at 12.
### 6. llama.cpp version watch
b10433 (2026-08-15) is adequate. Through b10451 there are no
spec-decoding fixes that help us; DFlash's multi-slot corruption is still
open upstream (fix #27117 pending). If that lands, DFlash/DSpark become
re-testable candidates (DSpark was the best single-machine draft before the
ROCm crash). No rebuild planned until then.
## Quantization landscape (reference)
The constraint is to keep UD-Q8_K_XL. This section records what exists so
future decisions are informed.
| Format | Size | Bits/weight | Notes |
|--------|------|-------------|-------|
| **UD-Q8_K_XL (unsloth)** | 31.5 GB | ~8.5 (mixed) | **Current.** Dynamic mixed-block; bundled MTP head; our baseline. |
| Q8_0 (ggml-org, bartowski) | 29.1 GB | 8.0 | Uniform 32-block. Same-bit A/B candidate (lever 3). |
| UD-Q6_K_XL (unsloth) | ~22 GB | ~6.5 | Below the 8-bit floor; documented, not recommended under the constraint. |
| Q6_K | ~21.5 GB | 6.6 | Same tier as above. |
| **ROCmFP4 "STRIX" (ROCmFPX fork)** | 13.3–13.8 GB | 4.25–4.39 | Strix-specific NVFP4-style format, ROCm backend only, requires a forked llama.cpp. Published: 30.3 t/s vs 10.7 for Q4_K_M (2.83×) at 8k with MTP n_max≈4; PPL ≈ Q4_K_M. **This is the "fast path" — and it is a quant downgrade plus a fork dependency that breaks the Vulkan production path.** Kept here as the escape hatch if the 8-bit speed target cannot be met. |
| NVFP4 (Blackwell) | ~14 GB | 4.5 | Hardware path for RTX 50-series. Irrelevant to 3090/Strix. |
If the 8-bit PP result (lever 1) is disappointing, the decision order is:
push PP split/latency tuning → ROCmFP4 as a *second* service (A/B, not a
replacement) → revisit the constraint with Frank.
## Morning test matrix (2026-08-16)
Policy: stop both production services, run tests, restore. Each config =
one head restart (~10 min load incl. weight stream; ~2 min with the worker
tensor cache warm).
| # | Config | Measures |
|---|--------|----------|
| A | PP split 1,1, ROCm head, MTP n_max=4 + ngram | baseline PP |
| B | A + `SPEC_NONE=1` | raw (no spec) PP speed |
| C | A but head build `build-vulkan-rpc` (Vulkan side) | backend A/B |
| D | A + `--spec-draft-model` Q4_0 MTP file | draft-head A/B |
| E | A with n_max 3 and 5 | n_max curve on the split |
| F | A, plain Q8_0 model file | same-bit A/B |
| G | PP default proportional split (no `--tensor-split`) | confirms the 20/80 handicap |
| Ref | production frank-dragoon (Vulkan, 2 slots) | the bar to beat |
Per config: `pp-test.sh bench 4` (4k prompt / 256 out) and `bench 16`
(16k / 256). Report: decode t/s (server print_timing), prefill t/s, MTP
acceptance, worker VRAM, head GTT usage.
Success bar: **≥ 30 t/s at 4k and ≥ 20 t/s at 16k decode**, against the
production @16k bar in the Baseline section, with the same 8-bit model.
## Sources
- GGUF header dump of the production model (this machine, 2026-08-15).
- Qwen3.8-27B model card + `ggml-org/Qwen3.8-27B-GGUF` /
`unsloth/Qwen3.8-27B-GGUF` repo listings (Hugging Face).
- kingjones `Qwen3.8-27B-ROCmFP4-GGUF` Strix tuning card (MTP n_max curve,
Q4_0 vs Q8_0 draft head, ROCmFP4 numbers).
- visorcraft `strix-halo-llm-perf` (Vulkan tg / ROCm prefill, ICD pinning,
`-dio` for RPC).
- llama.cpp b10433 source: `src/llama.cpp` (device selection),
`common/speculative.cpp` (MTP/ngram-mod), `ggml/src/ggml-rpc` (protocol).
- Local benchmarks: the measurement record lives on [[frank-dragoon]] (Eras and
Laws) and in [qwen3.8-spec-matrix](qwen3.8-spec-matrix.md).
serverAppConfigs-docs/LLM/swift15-campaign-worklog.md
-72
View File
@@ -1,72 +0,0 @@
# Swift1.5 w4dn8 Campaign — Work Log
Campaign branch `swift15-radiance-w4dn8`. Goal: convert
`ukisai/Swift1.5-Qwen3.8-Flash-Next` (BF16, 335.3 GiB) to
`qwen4exp-swift15-w4dn8-i8-hc8m.rad` (GPTQ gate_up w4nl64 + block-fp8 down +
10 protected bf16 experts + int8 trunk + fixed-scale PLE) and swap it into
frank-archon on the 1.0.13 pin. Design settled (25-question grill); this log
carries execution facts only. The cutover runbook
`frank-archon-radiance-swift15-w4dn8.md` replaces this page at cutover.
## 2026-10-06 evening
- Purge freed 835.8 GB; Swift1.5 BF16 downloaded 131/131 shards (sha256 ok).
- Corpora built: calib 10,014,932 tok / 1,294 prompts; KLD 1,800,816 tok /
197 prompts (`/home/dank/corpora/out/`).
- Engine verdict: 1.2.0 REJECTED for serving (decode 29.21 vs control 30.08,
accept 0.7123 vs 0.7549 deterministic; vram-pair translation ledger-exact).
**Serving pin stays 1.0.13** `sha256:4c3f48…c322e`.
- bf16 rad-convert done: `/home/dank/models/rad/qwen4exp-swift15-bf16.rad`
(335.28 GiB, 248,077 vocab tokens baked).
- **Blocker resolved — engine-version probe law:** the calibration tap lives
in the architecture plugin `/opt/radiance/share/radiance/architectures/
qwen4exp_fp8.so`, NOT the radiance binary, and the image has no
`strings(1)` (greps return 0 for everything — positive controls
mandatory). 1.0.13 plugin: `RADIANCE_CALIB_DIR` only; 1.2.0 adds
`RADIANCE_CALIB_TOKENS`. Both `libquant.so` carry the gptq quantizer
(`calib=/damp/act_order/cd`) and the identical `RADGRAM2` format
(magic + `gram.r%d.%s.bin` + calib_gram/calib_resolve/calib_stem).
**Decision: calibrate on 1.2.0, final convert on 1.0.13** — the serving
pin produces the served `.rad`; only the Gram files cross versions.
- `run-swift15-bf16.sh` boot-failure trail (all fixed, committed):
duplicated `radiance` entrypoint arg (rc=2); ratio-derived KV pool SHORT
217 MiB at 32×16k (explicit `--vram-kv-mib`); explicit 4608 starved the
disk tier — **weights slab = card − headroom − kv, disk-tier floor
9.57 GiB** (4.21 layer buffers + 5.36 fixed) → kv 3072; `KLD_MOUNT`
referenced-never-defined emptied the corpus mount.
- KLD record launched on 1.0.13 (disk tier, kv 3072, 4×16k, MNBT 32768):
measured ~116 tok/s at the 32k step (44 docs/58 min).
- Chain driver `kld-then-calib.sh` (pid 2848206, log
`/home/dank/logs/swift15-chain.log`): KLD rc=0 → verify kld-ref → boot
1.2.0 calib (MNBT 65536/seqs 8 per MOE-W4 930 tok/s law, synced commit
45de2fd) → `calib-push.py` parallel 16 → kill → prod restored by trap.
- Ready on the branch: `convert-swift15-w4dn8.sh` (1.0.13 convert, gates:
≥126,000 MiB), `apply-protected-cast.sh` (derive → recipe PROTECTED
section, refuses double-derive), `kld-gate-swift15.sh` (candidate vs
reference on 1.0.13, gate mean KL ≤ 0.03 / top-1 ≥ 95 %).
- `frank-archon` showing `failed` during the campaign is the 143-on-stop
signature of the foreground podman unit — cosmetic, not a crash.
## 2026-10-07 morning
- **KLD reference COMPLETE** on 1.0.13: `/home/dank/corpora/kld-ref` 24 GiB,
`radiance-kld-1`, `rows: 50235`, 197 docs, top-16 log-probs (f16),
corpus fnv1a64 `a9d62b33c18b3d9c`. Record took ~4 h at the 32k step.
The wrapper died rc=127 AFTER the engine exited clean: the mid-run scp of
commit 45de2fd shifted bash's lazy read offset — script-swap law, third
fleet occurrence. Resume path added: `SKIP_KLD=1` + `rows: 50235` receipt
gate (cf6c62d). **Never scp a script a running process owns.**
- Calibration chain3 live on 1.2.0 (big-step shape MNBT 65536/seqs 8): push
started, requests landing at 7–8k tokens.
## Campaign order after the chain
1. derive protected experts from Swift's own grams; apply to recipe.
2. Delete bf16 `.rad` (335 GiB) to `.AGENTTRASH` staging (Freed space gates the convert).
3. `convert-swift15-w4dn8.sh` (~2 h, CPU-only, prod can stay up).
4. rad-tune declare TP=2 spec=2; vram-pair boot arms; bench (gate decode ≥ 30.8 t/s).
5. `kld-gate-swift15.sh` (fail → escalate gate_up w4→w5nl, pre-authorized).
6. Cutover: docs, tests + e2e, deploy, probes. Throughput fallback: w4 down tier (Q35).
Disk ledger at KLD-record time: 104 GiB free → KLD ref −50 → Grams −20 →
bf16 delete +335 → convert +132 written.
serverAppConfigs-docs/LLM/ubatch-benchmark-large.md
-56
View File
@@ -1,56 +0,0 @@
---
type: benchmark plan
title: frank-dragoon ubatch sweep
description: Standardized ubatch sweep for the active kingjones Qwen3.8 ROCmFPX deployment.
tags:
- llama.cpp
- rocm
- benchmark
- ubatch
timestamp: 2026-09-01
---
# frank-dragoon ubatch sweep
This benchmark uses the active `frank-dragoon` model on Dankclaw with
`llama-bench`. It does not use `llama-server`, HTTP requests, or server-log
scraping. Stop the production service before running it and start the service
again afterward.
The old custom harnesses are retained in `.AGENTTRASH/bench-scripts-20260901/`
for auditability only. Their results are invalid and must not be used.
## Sweep
The requested points are `256, 512, 768, 1024, 1280, 1536, 1792, 2048,
3072, 4096` at 8K context, with `batch` equal to `ubatch` at every point.
Each point runs three 128-token repetitions by default.
Run on Dankclaw from the repository checkout:
```bash
sudo systemctl stop frank-dragoon
./scripts/bench-ubatch.sh
sudo systemctl start frank-dragoon
```
The script uses the active Q6 AGENT model, an 8K prompt workload, equal
`batch` and `ubatch` values, and `llama-bench` JSONL output. The sweep is:
```text
256, 512, 768, 1024, 1280, 1536, 1792, 2048, 3072, 4096
```
The GPU is shared with live workloads; do not run with the production service
up.
## Results
No valid results are recorded here. All results from the retired custom
harnesses are explicitly discarded. Record the raw `llama-bench` JSONL output
from the new harness here after a clean run.
## See also
- [LLM fleet index](index.md)
- [Deployment](deployment.md)
- [[frank-dragoon]] — the kingjones MTP era this sweep ran against is an Eras row there (its page was folded in 2026-10-18)
serverAppConfigs-docs/LLM/vLLM Boot Forensics and Config Interrogation.md
-208
View File
@@ -1,208 +0,0 @@
---
type: reference
title: vLLM Boot Forensics and Config Interrogation
description: Capturing a vLLM container boot failure's real root cause from journald (the container log is already gone), classifying the failure, and interrogating a live endpoint's true config and client contracts.
tags: [vllm, forensics, journalctl, podman, config, contracts]
timestamp: 2026-10-05T00:00:00-07:00
---
# vLLM Boot Forensics and Config Interrogation
For serve-script vLLM endpoints (the vLLM eras of [[frank-archon]] — that page's
Eras table is which ones; any endpoint that runs `podman rm -f` + a restart
service).
## 1. Capture the true cause — the journal, not `podman logs`
The serve script runs `podman rm -f <container>` at every start, so a dead
container's log is GONE by the time you look; `no container with name or ID`
means the journal is the only source. The engine's stdout survives in
journald under the container tag:
```bash
journalctl --no-pager --since "20 min ago" | grep -a "<container>\[" | tail -40
journalctl --no-pager --since "8 min ago" | grep -a "<container>\[" \
| grep -aiE "error|exception|assert|raise|invalid|failed" | grep -av APIServer | tail -12
```
Why naive greps fail:
- Default format truncates lines at the display cut, and the fork's giant
single-line ERROR dumps are journald-SPLIT — the decisive sentence never
appears in short format.
- `-o cat` strips the `svc[pid]:` prefix, so grepping the tag against `-o
cat` returns nothing — use the default format for the tag, `-o json` for
the content.
- Time filters: relative (`--since "NN min ago"`) only — absolute timestamps
are local PDT while the engine logs UTC; never mix.
**The JSON decode procedure** (proven 2026-10-01, caught
`torch.OutOfMemoryError` exactly):
1. Bound the window: last traceback lines + the `container died` podman
event.
2. `journalctl --no-pager --since … --until … -o json | python3 -c …` on the
remote: handle `MESSAGE` possibly arriving as a **byte list**
(`bytes(int(x) for x in m).decode("utf-8","replace")`), strip ANSI
(`re.sub(r"\x1b\[[0-9;]*m", "", m)`), dedupe by first ~12 chars (one
logical traceback can arrive as 20+ entries).
3. Reassemble the worker traceback: filter skeleton lines
(`File |…| raise |^+|~+|…<N> lines|…|re-`), keep content lines. The
decisive line is usually the last content line.
4. **The diagnosis is the gap math: `ask − free = shortfall`.**
**The wrapper trap:** `RuntimeError: Engine core initialization failed. See
root cause above` is the WRAPPER — the real cause is the EngineCore traceback
above it, ending in `RuntimeError: Worker failed with error '<cause>'`. Grep
for `Worker failed with error`.
**For probe boots / runner scripts:** capture live BEFORE teardown —
`podman logs -f <ctr> > /home/dank/logs/<arm>.engine.log 2>&1 &` right after
launch. A runner's `podman rm` destroys the only copy; two 2026-10-01
failures had their root cause eaten this way, and an 8-line bootfail capture
is insufficient.
## 2. Failure classes
| signature | class |
|---|---|
| `CUDA out of memory. Tried to allocate X GiB … Y GiB free` | capacity; the gap math names the next rung ([[RDNA4 MoE Expert Offload VRAM Budget]]) |
| `Worker failed with error 'Constraints violated (L['query_start_loc'].size()[0])! … specialized it to a constant (N)'` | torch.compile/dynamo shape specialization (N = num_seqs+1-ish) — NOT memory |
| `Deactivated successfully` with no traceback | container exited CLEAN (vLLM API server exits 0 after an EngineCore fatal); container already removed — re-read the journal, check `NRestarts` |
| `no container with name or ID` | container rm'd; journal only |
Boot-order reality: the plan computes fine at impossible budgets; the
failure lands later in worker init, not at plan time. A plan line printing a
huge arena does NOT mean it will boot.
Bypasses this fork exposes for compile faults: `--enforce-eager` (skips
torch.compile AND CUDA graphs — drops graphs, a real cost),
`VLLM_DISABLE_COMPILE_CACHE=1`, `VLLM_USE_AOT_COMPILE=1`. A stale cache is
usually NOT the cause — verify before "fixing" it (`~/.cache/vllm` may hold
only `modelinfos`; each `podman run` gets a fresh container FS unless the
cache dir is mounted).
**Crash-loop signature (shape does not fit):** `Started … ~6.7 min later …
Deactivated successfully … restart`. Diagnose with `systemctl show -p
NRestarts` + per-attempt duration. `Restart=always` loops visibly;
`Restart=on-failure` stays SILENTLY DEAD (`podman wait` returned 0, systemd
saw a clean exit).
## 3. Observation hygiene during recovery
- A container being STOPPED still answers `podman logs`, so a watcher
grepping its log reports PRE-change values and looks like the fix failed.
Discriminate by container identity:
`podman ps --sort created -a --format '{{.ID}} {{.Status}} {{.CreatedAt}}'`
— require the container to post-date the arm start, then read only that ID.
- A leftover probe script whose tail does `podman rm -f <prod-container>`
fires against a freshly DEPLOYED engine. Signature: a previously-healthy
unit exits 0 (`Started → Deactivated`, code 0) — an external rm, not a
crash. Suspect an external actor before debugging the engine; `pgrep -af`
every script you launched (and kill strays of `deploy-remote.sh` by
explicit PID — killing the dev-side deploy job does NOT kill its host-side
runner).
## 4. Ops laws that bite during recovery
- `Restart=always` does NOT fire after an explicit `systemctl stop` — a
deploy's stop leaves the unit down until something starts it. Check
`systemctl is-active` yourself.
- `ExecStop=podman stop -t 120` requires `TimeoutStopSec` > 120; the 90 s
default kills the stop and logs `Failed with result 'timeout'`.
- A `D`-state worker cannot be reaped by ANY signal (SIGTERM/KILL/`podman rm
-f` all fail). Check `ps -o pid,stat,etime,comm` before declaring a GPU
wedge — measured 2026-10-01: D → R on its own within ~3 min, then the
container stopped normally.
- Never `pgrep -f`/`pkill -f` a pattern that appears in your own command
line — kill by explicit PID list.
- `git stash -u` (not bare `git stash`) for deploy windows — untracked files
still trip the deployer's dirty-tree gate.
- A production engine started by a transient/ladder unit is NOT
boot-persistent; pin the winning numbers in the repo and deploy the real
unit.
## 5. Logging audit (what more can this container say?)
Answer with evidence from the live image, not upstream docs:
1. Enumerate the fork's own env registry:
`podman exec <ctr> python -c "import vllm.envs as e; print(chr(10).join(k for k in dir(e) if k.isupper() and any(s in k for s in ('LOG','DEBUG','TRACE','STATS','PROFILER','VERBOSE'))))"` — then read defaults/clamps in `vllm/envs.py` (e.g. `VLLM_LOG_STATS_INTERVAL` defaults 10.0 s, clamped ≥ 1.0; telemetry emits only while requests are in flight).
2. **Find the REAL HIP/ROCm library paths before grepping for AMD env vars** —
containers often install ROCm into system dirs, not `/opt/rocm`
(`podman exec <ctr> sh -c 'find / -name "libamdhip64.so*"'`). A grep of
`/opt/rocm/lib` that finds nothing is a FALSE NEGATIVE, not an absence.
Then `strings <lib> | grep -E '^AMD_LOG_LEVEL$|^HSA_ENABLE_DEBUG$'`.
3. Scan vendor kernel `.so`s for switches — and separate tuning vars from
logging vars (both share a prefix).
4. **Measure the DEBUG payoff**: count `logger.debug` sites in the module of
interest. 0–10 sites ⇒ a per-module DEBUG bump buys almost nothing; global
`VLLM_LOGGING_LEVEL=DEBUG` additionally makes api_server log full
request/response bodies (incident window only).
5. Check the deployed argv for `--disable-log-stats` / `--log-configs` to
know what is already suppressed.
Report: lever / effect / cost table, split into permanent-safe (stats
interval), incident-window (`AMD_LOG_LEVEL`, DEBUG), and does-not-exist
(absent strings). Wire via env-gated `-e` pass-throughs so default-off
preserves behavior.
## 6. Interrogate a live endpoint's REAL config
Flags and env diverge from intent constantly — rollbacks drop them silently.
Read what the container is actually serving, then prove the client contract
behaviorally:
```bash
# live argv (the truth)
ssh host 'podman exec <ctr> bash -c "cat /proc/1/cmdline | tr \"\\0\" \" \"" | tr " " "\n" \
| grep -A1 -E "max-model-len|max-num-seqs|kv-cache|reasoning-parser|tool-call-parser|template-kwargs|utilization|speculative"'
# container env knobs (fork tuning that never appears in argv)
ssh host 'podman exec <ctr> bash -c "env | grep -E \"R4D_|CLAV_|VLLM_\" | sort"'
# capacity contract as measured at boot
ssh host 'podman logs --tail 3000 <ctr> 2>&1 | grep -aE "GPU KV cache size|Maximum concurrency|non-default args"'
```
`non-default args` lines print what the server overrode — the fast way to
catch a flag that never landed. `GPU KV cache size: N tokens, Maximum
concurrency for M tokens per request: X.XXx` is the capacity contract:
N ≥ requests × M or the last client gets truncated.
**Thinking contract:** a reasoning model on vLLM returns `reasoning` ONLY if
the server runs a reasoning parser; without it the thinking lands INLINE in
`content` with a dangling close-tag and a null `reasoning`. Verify with a
temp-0 completion: pass = `reasoning` non-empty AND no `<\/think>` in
`content`. Diagnose with `ReasoningParserManager.list_registered()`
(import `vllm.reasoning`).
**Tool contract:** `tool_choice: "auto"` on a server WITHOUT
`--enable-auto-tool-choice` returns HTTP 400 — reads like a model fault, is a
config gap. Probe with a real tool schema and check `tool_calls` +
`finish_reason`.
**Restoring a dropped contract from history:**
```bash
git log --all --oneline -S"reasoning-parser"
git show <commit>:scripts/serve-<svc>.sh | grep -B3 -A3 "reasoning-parser"
```
Qwen3-family values that work on the tcclaviger fork: `--reasoning-parser
qwen3`, `--enable-auto-tool-choice --tool-call-parser qwen3_xml`,
`--default-chat-template-kwargs '{"enable_thinking": true,
"preserve_thinking": true}'`. `preserve_thinking`/`enable_thinking` are
chat-template variables, NOT engine flags: they belong in
`--default-chat-template-kwargs` server-side or `chat_template_kwargs` in the
request body; a bare top-level body field of that name is ignored. Check the
payload's `tokenizer_config.json` carries the key before trusting it either
way.
`/health` 200 says the engine is up, not that the contract is on the flags
you think — any client report of odd content (inline thinking, 400 on tools,
clipping at depth) starts with sections 1–3 here before touching anything.
## Related
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan arithmetic behind OOMs
- [[frank-archon]] — deployed shape and history
- [[frank-dragoon]] and [[frank-zealot]] — probe/ladder hygiene on live hosts
serverAppConfigs-docs/LLM/vLLM Porting and Compile Triage.md
-330
View File
@@ -1,330 +0,0 @@
---
type: reference
title: vLLM Porting and Compile Triage
description: Forward-porting a model architecture to a new vLLM major (additive-only discipline, gap enumeration, shims that survive worker spawn) and the torch.compile/dynamo break triage, including the EAGLE3/DFlash2 aux-hidden-state protocol port rules.
tags: [vllm, porting, torch-compile, dynamo, eagle3, dflash, glm]
timestamp: 2026-10-05T00:00:00-07:00
---
# vLLM Porting and Compile Triage
## Part I — Forward-porting a model architecture
### 1. Check whether a port already exists; size the work honestly
Read what the target image already ships before assuming a port is needed: a
backport's upstream PR may be merged in the target version, and the target may
carry a native kernel path the older line lacks (e.g. `rfi_hip` for RDNA4
native quantized MoE). Porting around a native path that was already there is
the expensive mistake.
An upstream backport ships two kinds of file: **additive** (the new model
package, its config, its kernel modules — port these) and **whole-file
replacements of vLLM core** (`model_runner.py`, `scheduler.py`,
`kv_cache_utils.py`, platform files — these are version X's copies of files
version Y restructured; copying them is a DOWNGRADE, not a port). Measure both
(`find … | xargs wc -l`), and say the additive-only ratio out loud before
starting — one real case: a 71,724-line overlay reduced to 10,385 additive,
and the "missing" drafter was already registered in the target.
### 2. Enumerate the gap in ONE pass, not one ImportError at a time
Inventory the target tree first (`model_executor/layers/`,
`v1/attention/backends/`, quantization backends, spec-decode drafters in
`registry.py`, `third_party/flash_linear_attention/ops/`). Then walk the
staged package with `ast`, resolving every `from vllm… import X` against the
target, split into MODULE-missing vs NAME-missing.
- **The double-prefix bug:** the dotted module already starts with `vllm.`,
so joining onto the package dir needs
`rel = mod.split(".", 1)[1].replace(".", os.sep)`.
- **Static scans over-report:** bare annotations, TypeAlias and star
re-exports resolve at runtime. Do a runtime check
(`importlib.import_module` + `getattr`) before quoting numbers — one scan
said 151/163 resolved; runtime said 10/13.
- **Classify:** redirects dominate (version moved a symbol to a submodule) —
a ~60-line rewriter splitting `from X import (a, b, c)` blocks handles
them; **delete a block that empties** (a bare `from X import (` with an
empty body is a SyntaxError) and gate the rewriter on `ast.parse` of every
touched file.
- **Port absent symbols with AST extraction**, not copy-paste: locate the
`ClassDef`, check every base/decorator resolves in the target, append the
exact segment, re-`ast.parse` the target, roll back on failure, skip if
already defined.
### 3. Silent-default traps and transformers gates
- Check every `cfg_int(cfg, key, DEFAULT)` against the new checkpoint: if the
key was renamed, the DEFAULT silently survives and the model computes fluent
nonsense with no error. Diff every config key the old parser reads against
the new `config.json`. When the old port used a fallback, the CHECKPOINT's
value must win (measured: `num_experts_per_token` fallback 7 vs
checkpoint `num_experts_per_tok=8`).
- `model type X but Transformers does not recognize this architecture` =
pinned transformers predates the model. Upgrading transformers inside a
pinned vLLM image breaks things hard to attribute — prefer registering a
genuine `PretrainedConfig` subclass:
`CONFIG_MAPPING.register("glm5_next", Glm5NextConfig)`, gated by a real
round-trip (`AutoConfig.from_pretrained(<ckpt>)` returns the right class).
If the target image's transformers is below vLLM's declared minimum,
upgrading inside the image beats a shim.
- **Hardware-gate flags from the port's own docs**: search the README for the
arch name and "do not" — e.g. a native quant MoE backend that is
CUDA/CDNA-gated falls back to per-call quantize-dequantize emulation at
~5x slower; pin those flags in the serve script with a comment citing the
line.
- **Backend selection:** default to `auto`, then READ the chosen-backend
line. `TRITON_UNFUSED` rejects some activation keys; `TRITON_ATTN` rejects
MLA + sparse attention — both surface as an init rejection. **"Using
'EMULATION' backend" is a FAILED configuration, not a success** (weights
dequantized per call: memory + throughput cost).
- `Loading model from scratch…` for many minutes usually means the port
SUCCEEDED and you are in weight placement — verify all layers constructed,
both TP workers alive, zero errors, before chasing memory. And if it does
not fit, **prove the band is empty**: sweep the knob and show no setting
works; a single OOM is not a conclusion. Keep `MemAvailable` guarded
(abort on low RAM; do not let the OOM killer take co-tenants).
### 4. Shims that must survive worker spawn
vLLM v1 workers are spawned processes — a parent-process monkeypatch is gone
by weight-load time. In order of preference:
1. **Patch the installed library's source file on disk** (reliable). Locate
the real install path at runtime:
`python -c "import transformers,os;print(os.path.dirname(transformers.__file__))"`.
2. `.pth` in site-packages — works but runs at interpreter start and can
import a package mid-import (circular import through `accelerate`);
expect failure.
3. `sitecustomize.py` — often silently DEAD: Debian ships its own at
`/usr/lib/python3.12/sitecustomize.py`. Verify it is actually imported
before debugging why it has no effect. (A `.pth` on `PYTHONPATH` is never
executed by `site.py`; `sitecustomize.py` is.)
Always confirm with a FRESH process — never with the process that set the
patch up (`is_moe`, `num_experts_per_token`, `mhc_num_residual_streams`).
**Misleading OOM names on ROCm:** `torch.AcceleratorError: CUDA error: out of
memory` from `pin_memory()` is usually the HOST `memlock` rlimit, not VRAM.
Cannot be raised from inside a container or root's own shell; only PAM at
login: `/etc/security/limits.d/*.conf` + a NEW session. Confirm by pinning
2/8/16 GiB — if those succeed, the rlimit was a real bug but is not the
failure you are chasing.
### 5. Ship as an idempotent patch-at-start applier
One `apply.sh` at container start, each step guarded by a grep-for-marker
(re-entry safe), ending in a verification block that imports every installed
symbol and prints PASS/FAIL per symbol — ~10 s instead of a multi-minute
load, and it turns "I think it works" into evidence.
---
## Part II — torch.compile / dynamo bring-up triage
### The five laws (measured on torch 2.11+rocm, vLLM 0.29-dev)
1. **`suppress_errors` does NOT cover graph breaks** — fullgraph=True +
suppress_errors=True still raises `Unsupported`. Only `fullgraph=False`
tolerates a break.
2. **A warm `lru_cache` is not trace-safe** — dynamo re-traces the wrapped
body to build its guard even on a hit ("Dynamo ignores the cache wrapper
and directly traces the wrapped function"). Resolve the value ONCE into a
plain module global and read that; a global read folds, a cached call does
not. Verified in-image: warm lru_cache-calling function still fails
fullgraph=True; a plain global read compiles.
3. **AOT compile refuses breaks outright** ("Graph breaks are not supported
with aot compile") — set `VLLM_USE_AOT_COMPILE=0` for a break-tolerant
run.
4. **Fatal vs tolerated breaks:** tolerated breaks get a resume function
(untraceable part runs eagerly, numerics unchanged). Fatal ones cannot:
`Unsupported: Read uninitialized cell` (a traced callee closes over a name
bound after its definition) and `Skip calling torch.compiler.disable()'d
function` under fullgraph=True. `torch.compiler.disable` is the documented
escape hatch for host-side/stateful work (ContextVar reads, lazy
allocation, workspace grab) and is a NO-OP when nothing compiles — safe to
ship in the stock path.
5. **Trust the frame chain, not the line number** — dynamo INLINES user
callees; an inlined failure reports the enclosing function's `def` line.
"KeyError in forward at the def line" is really "a callee raised".
`TORCHDYNAMO_VERBOSE=1` adds nothing for a plain Python exception (only
for dynamo-internal errors).
**The tell** that the trunk lacks compile support at all:
```
Cudagraph mode FULL_AND_PIECEWISE requires piecewise capture, but the loaded model
provides neither a compiled submodule nor breakable CUDA graphs. Overriding to
FULL_DECODE_ONLY.
```
Diff against the family that DOES compile and copy its
`@support_torch_compile` decorator including `dynamic_arg_dims` and any
`enable_if` hook; env-gate every change so the file can stay mounted for an A/B.
### Procedure
0. **Establish what configuration is ACTUALLY in effect**:
`podman exec <ctr> printenv VLLM_COMPILE_FULLGRAPH VLLM_USE_AOT_COMPILE …`.
A launcher that "sets" an env var in a branch may silently drop it — four
consecutive boots ran `fullgraph=True` while the operator believed
otherwise, which makes the reported failure MOVE as breaks are removed and
invites entirely wrong theories.
1. **Boot with the break list, not the first error:** `TORCH_LOGS=graph_breaks`
+ `VLLM_USE_AOT_COMPILE=0` + `VLLM_COMPILE_FULLGRAPH=0`. Re-verify each
candidate break under the flag-free config you actually intend — a break
that appears only under the logging flag is a diagnostic artifact.
2. **Fix in priority order:** (a) host syncs in the traced region
(`.item()`/`.cpu()`/`.tolist()` — check reachability first, a dead fallback
branch is not your blocker); (b) capability probes doing I/O in `forward`
(`os.path.exists`, `.so`/`ctypes` lookups — resolve once into globals,
warm at import); (c) lazy stateful work (hoist, or `torch.compiler.disable`
a thin wrapper); (d) new kernel specializations — check the vendor ships
SOURCES (`nm -D`/`strings`): a binary-only templated `kernel<128|256|512>`
makes the 288 you need unreachable. Fix at the PATTERN level (one
launcher/decorator family breaking all 39 call sites), not site level.
3. **KeyError on a bare module name** = dynamo's name→object map miss
(`OutputGraph.nn_modules`: registered `output_graph.py:1259`/`:1384`,
read back `:3000`, `_buffers/_parameters/_modules` stripped at `:1225`).
Ship a probe via `sitecustomize.py` that patches
`OutputGraph.example_value_from_input_node` and keeps weakrefs of live
graphs, printing `node.target`, key count, near-miss keys — separates a
cross-graph reference from a name-derivation mismatch.
4. **Instrumenting a startup hook is a FIREARM — arm it once:**
```python
if os.environ.get("PROBE_ARMED") == "1":
pass # an ancestor already armed it -> stay inert
else:
os.environ["PROBE_ARMED"] = "1"
...install probes...
```
A `sitecustomize.py` on `PYTHONPATH` imports in EVERY interpreter and
children inherit `PYTHONPATH`: an unguarded hook that imports the engine
self-replicates (observed: 19+ levels, 33 processes alive 18 min later).
Kill the chain by removing the container whose cgroup owns every
generation, then verify with a precise `/proc/<pid>/environ` count —
NEVER pattern-kill an interpreter name on a host running python in
production containers. Also chain the image's own sitecustomize so its
hook survives.
5. **Cheap in-image classification (no boot):**
`g = torch.compile(fn, backend="eager", fullgraph=True)` inside the
container (with `--device /dev/kfd --device /dev/dri` — torch.compile
queries a device even on the eager backend). ~5 s per question instead of
~7 min boots.
6. **Package as real patch artifacts:** hand-written `@@` headers are not
diffs and `patch(1)` rejects them. Generate
`diff -u --label a/<path> --label b/<path> orig new > name.patch` with
`--- a/` / `+++ b/` paths so `-p1` works; verify by applying to a fresh
extraction of the pinned image and comparing sha256 (`sha256sum` must
equal the `b/` hash recorded in the header). Header carries: STATUS, the
measured evidence, base+patched sha256, how to apply, the NEXT blocker.
### Cost model before investing
**Profile first** — parse the rank0 chrome trace (`POST /start_profile`,
device-activity rows) for the kernel table. Measured case: the long-suspected
expert-gather was **1.05 %** of device time while **44 %** was unfused
pointwise glue (708,500 `vectorized_elementwise_kernel` launches =
1,428/token) — a months-old "structural" conclusion died in one afternoon.
Speculative decoding does NOT help a shape whose per-token cost scales with
the token count, and it hurts prefill. Decide the ceiling up front: if the
fused target is ~13–15 tok/s, do not spend a night of boots chasing 35.
---
## Part III — EAGLE3 / DFlash2 / DSpark aux-hidden-state protocol port
Goal: make a trunk satisfy `SupportsEagle3` so a DFlash2 drafter
(`DFlash2DraftModel`, method `dflash`) boots. Built on the GLM-5.3-Flash port
(tcclaviger fork, 2026-09-30; four failed boots produced the laws below).
### The four laws
1. **Patch the instance the runner calls.**
`set_eagle3_aux_hidden_state_layers(self.model)` (model_runner.py, right
after `model_loader.load_model`) runs on the LOADED instance — for a
multimodal model that is the `…ForConditionalGeneration` WRAPPER (the
APIServer log line `Resolved architecture: …` names it), NOT the text
model it holds as `self.language_model`. Patching only the inner
`…ForCausalLM` reproduces the identical `RuntimeError: Model does not
support EAGLE3 interface` forever. Patch BOTH wrappers.
2. **The check is Protocol isinstance, NOT duck typing.**
`supports_eagle3(model)` = `isinstance(model, SupportsEagle3)`
(interfaces.py ~1728; caller `v1/worker/gpu/spec_decode/eagle/eagle3_utils.py:20`).
`SupportsEagle3` is `@runtime_checkable` over base `SupportsEagleBase`
(`:1486`, `:1659`); `typing._ProtocolMeta.__instancecheck__` does
`getattr_static` over ALL `__protocol_attrs__` — including BASE-protocol
members — and breaks on the FIRST miss. Full member set:
`supports_eagle3`, `has_own_lm_head`, `has_own_embed_tokens`,
`set_aux_hidden_state_layers`,
`get_eagle3_default_aux_hidden_state_layers`. **Class attributes count as
presence; a methods-only implementation FAILS** because the three attrs
are missing. Verified in-image with stock typing, no boot: flag-only →
False; flag + two base members → True.
3. **MRO forbids inheriting the Protocol.** Appending `SupportsEagle3` to a
wrapper already carrying Protocol-derived bases dies at class creation:
`Cannot create a consistent method resolution order (MRO) for bases …` —
MRO simulation (stub Protocols + `type("C", bases, {})`) confirmed no
position works. This failure also PROVES the isinstance-semantics question
is moot: class creation precedes any check.
4. **Write the two methods by hand** — without inheritance the Protocol
defaults do not exist. Text model: unwrap to the trunk
(`self.model.aux_hidden_state_layers = layers`); multimodal wrapper:
delegate to `self.language_model` (the pixtral.py:498 pattern).
`get_eagle3_default_aux_hidden_state_layers` returns `(2, L//2, L-3)`; the
drafter config's `target_layer_ids` wins when present (GLM DFlash2 ships
`[5, 14, 24, 33, 42]` of 45).
The fix shape (on EACH wrapper class):
```python
supports_eagle3: ClassVar[Literal[True]] = True
has_own_lm_head: bool = False # False when embed/lm_head are delegated
has_own_embed_tokens: bool = False
def set_aux_hidden_state_layers(self, layers): ...
def get_eagle3_default_aux_hidden_state_layers(self): ...
```
### Trunk mechanics
The trunk (`Glm5NextModel`) must be `EagleModelMixin`; add
`aux_hidden_state_layers: set[int]` (default empty) + `start_layer` handling.
Capture aux states in the layer loop: the state ENTERING each requested
layer, BEFORE its input norm (GLM's loop carries the single-stream state;
deepseek_v2's equivalent is `hidden_states + residual`).
`use_aux_hidden_state_outputs=True` is set in model_runner.py (~:287) for
methods `dflash`, `dspark`, `extract_hidden_states`; the forward returns
`(hidden_states, aux_hidden_states)` conditionally after `self.norm` (tuple
consumed at `v1/worker/gpu/cudagraph_utils.py:621`).
### Window discipline and verification
- Scratch boots on dankserver MUST stop + runtime-mask production
`frank-archon` first — the expert-offload planner reads `MemAvailable` at
plan time, and beside-prod boots die at plan with
`ExpertPlanError: host holds N of M swappable experts`. After stop+mask,
loop until `MemAvailable ≥ 230 GiB` before launching (a stop against a
mid-load engine SIGKILLs at ~90 s and the freed ~170 GiB takes >1 min to
return — a new window started during restore inherits the false failure).
- Never launch a window while the previous window's RESTORE is in flight
(`systemctl start` returns immediately; the engine loads ~6 min).
- Save full probe logs BEFORE teardown (`podman logs <probe> > full.log`);
decision greps truncate lines at `…` and tracebacks die with the container.
- Cheap verification, no boot: AST-parse the patched file for class bases +
member presence; MRO simulation with stub Protocols; in-image `typing`
replication of the isinstance walk (pure stdlib — importing vllm itself
crashes on GPU-less inspection). Boot check: the worker line `Using Eagle3
auxiliary layers from config: (…)` = gate cleared.
- Honest expectations: a target-side DFlash2/EAGLE3 port on a
pointwise-glue-bound trunk (~44 % of kernel time) buys ~ZERO decode gain —
the verification pass costs ∝ its token count. Port it for capability, not
speed.
## Related
- [[vLLM Boot Forensics and Config Interrogation]] — capture and contracts
- [[RDNA4 Container Serving]] — the serving stack being ported onto
- [[frank-archon]] — the live deployment
- [[frank-dragoon]] — the retired w4b-era endpoint and its profiling notes
serverAppConfigs-docs/LLM/vllm_research_findings.md
-226
View File
@@ -1,226 +0,0 @@
---
type: reference
description: "vLLM v0.26.0 (latest stable, released July 25, 2026) supports DFlash speculative decoding with the `--spec-method dflash` and `--spec-model` flags."
tags:
- llm
- vllm
timestamp: 2026-08-28
---
# Research: vLLM Parameters for Specified Configuration
## Summary
vLLM v0.26.0 (latest stable, released July 25, 2026) supports DFlash speculative decoding with the `--spec-method dflash` and `--spec-model` flags. KV cache dtype `q8_0` is **not supported** — only fp8 variants and turboquant options are available. vLLM does not have a dedicated MoE expert CPU offload flag; general CPU offloading via `--cpu-offload-gb` is the closest option. The `--max-num-seqs` flag is confirmed. Podman GPU access requires `--gpus all` with the NVIDIA container toolkit.
## Findings
### 1. Speculative Decoding with DFlash Draft Model
**Flag syntax:**
```bash
--spec-method dflash \
--spec-model z-lab/Qwen3.6-35B-A3B-DFlash \
--spec-tokens 8
```
- `--spec-method dflash` — DFlash is a first-class speculative method in vLLM v0.26.0. Defined in source as `DFlashModelTypes = Literal["dflash"]` in `vllm/config/speculative.py`.
- `--spec-model z-lab/Qwen3.6-35B-A3B-DFlash` — specifies the HuggingFace path to the DFlash draft model.
- `--spec-tokens N` — number of speculative tokens (block size). The DFlash model card recommends block size 8 for higher-concurrency serving, or 16 for concurrency-1 throughput.
- DFlash support landed in v0.25.0 (PR #46770, #46853) and was enhanced in v0.26.0 with hybrid SWA + full attention drafters (PR #47914).
- The DFlash draft model is **not standalone** — it must be paired with its target model (`Qwen/Qwen3.6-35B-A3B`).
**Additional speculative config options** (from `vllm/engine/arg_utils.py`):
- `--speculative-config '{"key": "value"}'` — full JSON config for advanced settings
- `--spec-tokens` — shorthand for `num_speculative_tokens`
**Caveat:** DFlash may require a non-causal-capable attention backend for the draft model. Set `--speculative-config '{"attention_backend": "FLASH_ATTN"}'` if needed.
[Source: vLLM v0.26.0 release notes](https://github.com/vllm-project/vllm/releases/tag/v0.26.0)
[Source: vllm/config/speculative.py source](https://github.com/vllm-project/vllm/blob/main/vllm/config/speculative.py)
[Source: z-lab/Qwen3.6-35B-A3B-DFlash model card](https://huggingface.co/z-lab/Qwen3.6-35B-A3B-DFlash)
---
### 2. KV Cache Dtype — q8_0 NOT Supported
**`q8_0` is NOT a valid KV cache dtype in vLLM.** The supported `CacheDType` values (from `vllm/config/cache.py`):
| Value | Description |
|-------|-------------|
| `auto` | Use model data type (default) |
| `float16` | FP16 KV cache |
| `bfloat16` | BF16 KV cache |
| `fp8` / `fp8_e4m3` | FP8 E4M3 (CUDA 11.8+) |
| `fp8_e5m2` | FP8 E5M2 (CUDA 11.8+) |
| `fp8_inc` | FP8 for Intel Gaudi (HPU) |
| `fp8_ds_mla` | FP8 for DeepSeek MLA |
| `turboquant_k8v4` | TurboQuant 8-bit K, 4-bit V |
| `turboquant_4bit_nc` | TurboQuant 4-bit non-causal |
| `turboquant_k3v4_nc` | TurboQuant 3-bit K, 4-bit V |
| `turboquant_3bit_nc` | TurboQuant 3-bit non-causal |
| `int4_per_token_head` | INT4 per-token-head |
| `int8_per_token_head` | INT8 per-token-head |
| `fp8_per_token_head` | FP8 per-token-head |
| `nvfp4` | NVFP4 quantization |
| `nvfp4_4over6` | NVFP4 4over6 layout |
**Closest alternatives to q8_0:**
- `fp8` (alias for `fp8_e4m3`) — 8-bit FP8, closest to q8_0 in precision
- `int8_per_token_head` — 8-bit integer with per-token-head scales
- `turboquant_k8v4` — 8-bit K cache, 4-bit V cache
**CLI flag:** `--kv-cache-dtype fp8` (or any of the above values)
[Source: vllm/config/cache.py source](https://github.com/vllm-project/vllm/blob/main/vllm/config/cache.py)
---
### 3. CPU Expert Offloading for MoE Models
**vLLM does NOT have a dedicated `--cpu-offload-gate` or `--expert-offload-cpu` flag.**
Available MoE-related flags (from `vllm/engine/arg_utils.py`):
- `--enable-expert-parallel` — enables expert parallelism (distributes experts across GPUs)
- `--expert-placement-strategy` — controls expert placement strategy
- `--moe-backend` — selects the MoE kernel backend
Available CPU offloading flags:
- `--cpu-offload-gb N` — general CPU offloading of N GB of model parameters (UVA offloading)
- `--cpu-offload-params "pattern"` — offload specific parameter patterns to CPU
- `--kv-cache-memory-bytes` — controls KV cache memory allocation
**How to offload MoE experts to CPU:**
Use `--cpu-offload-gb` with `--cpu-offload-params "experts.*"` to offload expert weights:
```bash
--cpu-offload-gb 32 \
--cpu-offload-params "experts.*"
```
This uses vLLM's UVA (Unified Virtual Addressing) offloading mechanism. The experts remain on CPU and are transferred to GPU on-demand during MoE routing.
**Caveat:** CPU offloading MoE experts will significantly impact latency due to PCIe transfer overhead. This is a memory-saving measure, not a performance optimization.
[Source: vllm/engine/arg_utils.py source](https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py)
---
### 4. max-num-seqs Flag — Confirmed
**Flag:** `--max-num-seqs N`
- Confirmed in `vllm/engine/arg_utils.py`: `max_num_seqs: int | None = None`
- Limits the maximum number of concurrent sequences (requests) in the scheduler
- When `None` (default), vLLM auto-calculates based on available KV cache memory
- Setting this constrains concurrency regardless of available resources
[Source: vllm/engine/arg_utils.py source](https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py)
---
### 5. Podman + vLLM + CUDA GPU Access
**Exact podman run flags for GPU access:**
```bash
podman run -it \
--gpus all \
--device /dev/nvidia0 \
--device /dev/nvidiactl \
--device /dev/nvidia-uvm \
--device /dev/nvidia-uvm-tools \
--device /dev/nvidia-modeset \
--cap-add=SYS_ADMIN \
--security-opt apparmor=unconfined \
-v /path/to/hf-cache:/root/.cache/huggingface \
vllm/vllm-openai:latest \
vllm serve <model>
```
**Minimal form (with NVIDIA Container Toolkit installed):**
```bash
podman run -it --gpus all vllm/vllm-openai:latest vllm serve <model>
```
**Key flags:**
- `--gpus all` — exposes all GPUs to the container (requires `nvidia-container-toolkit`)
- `--device /dev/nvidia*` — manual device passthrough (fallback if `--gpus` doesn't work)
- `--cap-add=SYS_ADMIN` — needed for NVIDIA driver access in some Podman configurations
- `--security-opt apparmor=unconfined` — disables AppArmor which can block GPU access
**Prerequisites:**
- `nvidia-container-toolkit` must be installed on the host
- Podman's `containers.conf` should have `nvidia` in the `devices` section, or use `--gpus all`
- NVIDIA driver must be installed on the host (not in the container)
[Source: Podman GPU documentation](https://podman.io/docs/gpu-support)
---
### 6. Latest Stable vLLM Version
**Version: v0.26.0** (released July 25, 2026)
- 411 commits from 212 contributors (61 new)
- Key features: DFlash speculative decoding, Model Runner V2 as default, DeepSeek-V4 optimizations, KV offloading improvements
- Available on PyPI: `pip install vllm==0.26.0`
- Docker image: `vllm/vllm-openai:latest`
[Source: PyPI vllm page](https://pypi.org/project/vllm/)
[Source: GitHub releases](https://github.com/vllm-project/vllm/releases)
---
## Complete Example Command
Putting it all together for a Qwen3.6-35B-A3B model with DFlash speculative decoding:
```bash
podman run -it \
--gpus all \
--device /dev/nvidia0 \
--device /dev/nvidiactl \
--device /dev/nvidia-uvm \
--device /dev/nvidia-uvm-tools \
--cap-add=SYS_ADMIN \
--security-opt apparmor=unconfined \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:latest \
vllm serve Qwen/Qwen3.6-35B-A3B \
--spec-method dflash \
--spec-model z-lab/Qwen3.6-35B-A3B-DFlash \
--spec-tokens 8 \
--kv-cache-dtype fp8 \
--max-num-seqs 32 \
--tensor-parallel-size 1 \
--gpu-memory-utilization 0.9
```
## Sources
### Kept
- [vLLM v0.26.0 Release Notes](https://github.com/vllm-project/vllm/releases/tag/v0.26.0) — authoritative release information
- [vllm/engine/arg_utils.py](https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py) — complete CLI argument definitions
- [vllm/config/cache.py](https://github.com/vllm-project/vllm/blob/main/vllm/config/cache.py) — CacheDType enum and CacheConfig
- [vllm/config/speculative.py](https://github.com/vllm-project/vllm/blob/main/vllm/config/speculative.py) — SpeculativeConfig and method types
- [z-lab/Qwen3.6-35B-A3B-DFlash](https://huggingface.co/z-lab/Qwen3.6-35B-A3B-DFlash) — DFlash model card with usage examples
- [PyPI vllm](https://pypi.org/project/vllm/) — version history
### Dropped
- Generic vLLM overview pages (GitHub README, docs landing page) — no specific parameter details
- Wikipedia vLLM article — not authoritative for CLI flags
## Gaps
1. **DFlash attention backend requirements** — The exact attention backend configuration needed for DFlash with Qwen3.6-35B-A3B on specific GPU architectures (e.g., H100 vs A100) was not fully verified. May need `--attention-backend flash-attn` or `flashinfer`.
2. **MoE CPU offload performance impact** — No benchmarks found for CPU-offloaded MoE experts in vLLM. Latency impact is expected to be significant but unquantified.
3. **Podman GPU on specific distros** — Exact Podman GPU flags may vary by distribution (RHEL vs. Fedora vs. Arch). The flags above are the most universal configuration.
4. **DFlash + MoE compatibility** — Whether DFlash speculative decoding works correctly with MoE target models (like Qwen3.6-35B-A3B) was not explicitly tested in the release notes.
## Residual Risks
- DFlash may require specific attention backends not available on all GPU architectures
- CPU offloading MoE experts will add significant latency; test thoroughly before production use
- `q8_0` is definitively not supported; using an unsupported dtype will cause startup failure
serverAppConfigs-docs/LLM/x4-pair-experiment.md
-409
View File
@@ -1,409 +0,0 @@
---
type: reference
title: The x4 pair — Flash-Next across the 3090 and the Strix Halo iGPU
description: What mainline llama.cpp can and cannot do with a 3090 + Strix Halo combination, measured 2026-09-16 on Dankclaw — the tensor inventory that decides any split, the single flag that kills per-tensor placement across devices (-ot), the two-device layer splits that DO work (pure-Vulkan and cross-vendor CUDA+Vulkan, +7 % decode at 64k depth), and the harness-parsing trap that made the first round of measurements meaningless.
tags:
- llm
- experiment
- dankclaw
- llama.cpp
- vulkan
- cuda
- strix-halo
- rtx3090
- flash-next
- pipeline-parallelism
timestamp: 2026-09-16
---
# The x4 pair — Flash-Next across the 3090 and the Strix Halo iGPU
The question this answers: **is there a 3090 + Strix Halo combination worth
building**, given that the only link between them is the 3090's PCIe Gen4 x4
(measured: `LnkSta: Speed 16GT/s, Width x4` under load, `Width x16` capable;
idle-downshifts to Gen1 x4).
Run 2026-09-16 with mainline llama.cpp `37b53fd`, a **CUDA + Vulkan dual-backend
build** (`build-dual`, `-DGGML_CUDA=ON -DGGML_VULKAN=ON`), both Dankclaw LLM
services stopped per the benchmarking policy.
## The tensor inventory decides everything
From the 111 GiB GGUF (`mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`,
1224 tensors, 176.94 B params, **A3B active**):
| group | size |
|---|---|
| `ffn_{gate,up,down}_exps` × 48 layers, 512 experts each | **75.0 GiB** |
| `per_layer_token_embd` — one PLE lookup table | **32.8 GiB** |
| everything else (attn q/k/v/qkv/o/gate, ssm × 36, hc, indexer, norms, output) | **3.2 GiB** |
- 48 layers, of which **12 are full attention** (indices `[3,7,11,15,19,23,27,31,35,39,43,47]`,
every 4th — `full_attention_interval = 4`) and 36 are GDN/linear.
- Only the 12 attention layers carry KV. At the engine's measured 28 KiB per
position, a 200k-token conversation holds **5.5 GiB of KV**.
- A layer split moves **whole layers**, so it moves ~1.62 GiB of experts with
every layer (75 GiB / 48). Attention layers are spread across the whole depth
(every 4th), so no contiguous layer range can hold them all.
The GGUF comes from `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF`, which
ships this quant as three `.partNofM` fragments (37.00 + 37.00 + 36.97 =
110.97 GiB) concatenated in numeric order. The same repo carries the smaller
rungs — i1-Q4_K_S 104.09 GiB (also as parts), i1-IQ4_XS 90.78, i1-Q3_K_L 90.02,
i1-Q3_K_M 87.67, i1-IQ3_M 83.39, i1-IQ3_S 82.78, i1-Q3_K_S 82.66, i1-Q2_K
74.92 GiB — which is where a "does a smaller quant buy anything?" test would
start (it buys memory, ~5 % decode, and costs quality; see the write-up below).
That is why the interesting split is *not* "put the experts somewhere else" —
experts are re-read once per token but they are only ~1.7 GB of traffic — it is
"get KV onto the 936 GB/s pool", because KV is the O(context) term.
## Results
`llama-bench`, same model. Shallow runs are `-p 512,16384 -n 64`; the
depth-matched pair is `-n 32 -d 65536`. Note llama-bench's `tg` instances run
with `n_prompt = 0`, so a plain `tg` number is **~zero depth**: `-d` is required
to put KV in the picture.
| config | placement | pp512 | pp16384 | tg (≈0 depth) | tg @64k depth |
|---|---|---|---|---|---|
| **A** | iGPU trunk + experts on CPU | 128.7 | 122.1 | 12.10 | — |
| **A\*** | iGPU trunk **and** experts (Vulkan1), PLE on CPU | **456.7** | **403.1** | **32.64** | **25.77** (pp512@64k 231.8) |
| **G2** | **pure Vulkan** layer split, iGPU 85 % / 3090 15 % | — | — | — | **27.55** (pp512@64k 224.3) |
| **H** | **cross-vendor** layer split, CUDA0 15 % / Vulkan1 85 % | — | — | — | 27.23 (pp512@64k **268.5**) |
| **B′** | trunk + KV on CUDA0, experts on CPU | 60.1 | 81.4 | 20.46 | — |
| **G1** | the design: trunk + KV on Vulkan0, experts on Vulkan1 (`-ot`) | — | — | **abort** | — |
| **B** | trunk + KV on CUDA0, experts on Vulkan1 (`-ot`) | — | — | **abort** | — |
| **B2/B3** | per-layer `-ot` / `-sm layer -ts` across CUDA+Vulkan | — | — | load failure | — |
Real placement in the split runs, sampled from `nvidia-smi` during the run:
**12,804–12,832 MiB** resident on the 3090 in G2 and **14,994 MiB** in H — the
split is genuine, not a mislabelled single-device run (see Finding 5 for why
that check is mandatory here).
Reference points, all era readings rather than measurements of this run: the
deployed halogen shape at ≈ **25 tok/s** at real contexts and the retired ik
hybrid at 3.8–4.6 tok/s (both are [[frank-dragoon]] eras, and that page's Eras
rows are where those shapes live); mainline with the trunk on the 3090 and CPU
experts at 1.01 tok/s (2026-09-15, this page's own earlier pass); the CPU expert
path's prefill ceiling at 192 tok/s.
## Finding 1 — per-tensor placement across devices is killed by one flag: `-ot`
The design (trunk + KV on the 3090, experts on the iGPU) needs
`--override-tensor` to place two *different* tensor groups on two devices. That
aborts in the scheduler:
```
ggml-backend.cpp:941: pre-allocated tensor (blk.0.ffn_down_exps.weight)
in a buffer (Vulkan1) that cannot run the operation (NONE)
```
The mechanism, read out of the source rather than inferred:
```c
// llama-context.cpp:429 — the scheduler only gets multi-device copy slots if:
bool pipeline_parallel =
model.n_devices() > 1 && ... &&
!model.has_tensor_overrides(); // ← line 433; a required term
// llama-context.cpp:605 — that flag is passed straight to the scheduler:
sched.reset(ggml_backend_sched_new(..., cparams.pipeline_parallel, ...));
// ggml-backend.cpp:25 — where it decides how many copies each tensor may have:
sched->n_copies = parallel ? GGML_SCHED_MAX_COPIES : 1;
```
With `n_copies == 1` the scheduler cannot give a pre-allocated (weight) tensor a
second home, so a weight whose buffer belongs to a non-primary device has no
backend and the graph build aborts. `has_tensor_overrides` is consulted in
**exactly one place in the whole tree** (llama-context.cpp:433), so no other
flag can rescue it.
- **The CPU is the only working target.** `-ot exps=CPU` runs because the CPU
backend is last in the scheduler's list and host buffers are always readable
(configs A and B′). A *device* target is what aborts.
- **Pure Vulkan does not change this.** Config G1 put both Vulkan devices
genuinely in the device list (verified by the correct `-dev` syntax, see
Finding 5) and aborted with the byte-identical message. The blocker is the
override/scheduler interaction, **not** a CUDA-vs-Vulkan driver mismatch.
## Finding 2 — two-device layer splits DO work, in both flavours
With no tensor overrides, `pipeline_parallel` stays on and the scheduler happily
runs a graph across two device backends. Two arrangements were measured at 64k
depth:
- **G2 — pure Vulkan** (`-dev Vulkan1/Vulkan0 -sm layer -ts 0.85/0.15`): iGPU
takes 85 % of layers plus the 32.8 GiB PLE table, the 3090 takes the last
7 layers (which include attention layers 43 and 47, the only KV-bearing layers
in that range). **27.55 tok/s** vs 25.77 for the iGPU alone: **+6.9 %**.
- **H — cross-vendor** (`-dev CUDA0/Vulkan1 -sm layer -ts 0.15/0.85`): the 3090
runs its layers on its **CUDA** backend, the iGPU on Vulkan. **27.23 tok/s**
and **pp512@64k 268.5** against 231.8 for the iGPU alone: same decode, **+16 %
prefill**.
So the pair *is* expressible — just not in the shape the design wanted.
## Finding 3 — the iGPU alone is competitive, and that was not expected
Config A\* — mainline llama.cpp, Vulkan/RADV, **everything on the iGPU**, PLE in
mmap page cache — delivers 32.6 tok/s shallow and **25.8 tok/s at a 64k-deep
context**, i.e. at fleet-realistic depth it *matches* halogen's deployed ~25
tok/s, with 456 tok/s fresh prefill (232 tok/s into a 64k cache) and the 3090
sitting at 0 %/39 MiB. Sampling during the run showed the iGPU at **99–100 %**
busy, so the work is genuinely there and not silently CPU-side.
The KV term is visible in exactly the predicted proportion: 32.64 → 25.77 tok/s
from 0 to 64k depth (−21 %), consistent with ~3 GiB of KV read per token at the
LPDDR's shared 256 GB/s.
**Why this is still not deployable here:** the run committed **116 of the host's
121 GiB** (78 GiB of Vulkan device buffers + 32.8 GiB of PLE page cache), leaving
5 GiB. Halogen serves the same model holding 95.4 GiB with 14.7 GiB to spare,
because it pages the n-gram table and pins only 68 GiB. Throughput parity, memory
bankruptcy: the fleet's ~28 co-tenant containers are the reason halogen's shape
exists. The layer split does relieve some of that pressure (the 3090 absorbs
~12.8–15 GiB of layer+KV) but it does not change the class of the result.
## Finding 4 — the split cannot reach the design's premise
The premise was "KV on the 936 GB/s pool". A layer split cannot deliver it:
- Moving an attention layer to the 3090 costs **1.62 GiB** (its MoE experts
travel with it — only whole layers move), so all 12 attention layers = 19.4 GiB.
- Add 200k of KV (9.2 GiB at f16, 4.6 at q8_0) and the card's ~24.1 GiB usable
is exceeded. Feasible only to ~64k context — precisely where the KV term is
small enough not to matter (the +7 % measured above is that residual).
- The 32.8 GiB PLE table must live on the *main* device (the first in the
`-dev` list, or whichever `-ts` share owns the output layer), so the iGPU keeps
a fixed 32.8 GiB no matter how the layers are split.
`-ot per_layer_token_embd=CPU` would move the PLE — but it is an override, and
Finding 1 applies: it turns multi-device scheduling off.
## Finding 5 — the harness-parsing trap that invalidated the first round
The first pass concluded "CUDA + Vulkan cannot serve one model". That conclusion
was an artifact. `llama-bench --help` reads:
```
-dev, --device <dev0/dev1/...> multi values separated by ','
-ts, --tensor-split <ts0/ts1/..> multi values separated by ','
```
`parse_devices_arg` splits each `-dev` value on `/` into a device *list*, and the
outer loop splits the argument on `,` into separate *sweep values*. So
`-dev CUDA0,Vulkan1` means **two single-device configurations** (CUDA0 alone,
then Vulkan1 alone) — never a two-device run. Consequences that looked like
findings:
- "the abort" — a single-device run with `-ot exps=Vulkan1` placing a weight on a
device that is not in the list at all (Finding 1's abort, for the trivial
reason).
- "load failure" for `-sm layer -ts 0.25,0.75` — `-ts 0.25,0.75` is *two sweep
values*, each becoming `[0.25, 0, 0, …]` / `[0.75, 0, 0, …]`, i.e. device 0
takes everything; the second config put 111 GiB on a 24 GiB card and OOM'd.
The tell that the runs were single-device: the bench table's own `dev` column
printed `Vulkan1` for every row. Four separate "iGPU-only" runs agree to within
0.2 % (25.71, 25.72, 25.77), which is exactly what an accidentally-single-device
sweep looks like.
**Rules that follow.** In llama-bench, `/` separates devices *within one
configuration* and `,` separates configurations to sweep; `-dev` and `-ts` both
follow this. And never trust a placement-dependent number without an independent
check of where the memory went — `nvidia-smi` sampled during the run is what
distinguishes a real split from a mislabelled sweep.
## Other landmines walked into (worth knowing before the next run)
- **`llama-cli` in conversation mode with EOF on stdin spams `> ` forever** —
3.2 GB in a few minutes, no error, no exit. Use `llama-server` for scripted
runs, or `-st/--single-turn` with llama-cli. (`-no-cnv` is not a flag in this
build.)
- **A dead rootless podman pause process takes the whole host's container
management down.** The first `invalid internal status` in the user journal is
**18:58:30**, from containers whose healthchecks shell out to the podman CLI —
i.e. the pause process died *while the last bench was still running*, not
after the runs finished. Every `podman` command for the user then failed with
`invalid internal status … could not find any running process`. The trigger is
**not** established: `journalctl -k` holds no OOM kill for that window, so the
tempting reading ("the 116 GiB runs squeezed it out") is a guess and is
recorded as one.
`podman system migrate` — the remedy podman itself names — **panics**
in 5.7.0 (`teardownNetworkBackend` nil deref). The rootless netns that holds
podman's bridge addresses is gone with it, so `aardvark-dns` cannot bind
`10.89.x.1:53` and *every* container start fails. Repair: reboot (it clears
`/run/user/<uid>` and the whole host comes back — 28 quadlets, `WantedBy`
present, linger on).
- **A `Network=host` service is not reachable from a bridge container at the
host's LAN IP.** netavark DNATs *published* ports only; a bare host-netns bind
is refused. Use `host.containers.internal` (169.254.1.2). This was found the
same evening when `hermes.franksdank.cloud` 502'd — fixed in the Caddyfile and
guarded by a strengthened seam (`tests/hermes_webui.bats`).
- **The memory budget includes the co-tenants' patience.** The iGPU runs held
116 of the host's 121 GiB for minutes at a time. Nothing in the kernel log
proves damage came of it, but running a model that big on a host that also
serves ~28 containers is asking the question — do it in a window, not
alongside.
## Conclusion
The fleet's answer is unchanged — **two independent endpoints**
([[frank-dragoon]] on the iGPU, [[frank-zealot]] alone on the 3090; each page
owns what its card holds) — but two claims from the first round of this
experiment are now corrected:
1. **Cross-vendor layer split is possible** (measured: CUDA0 + Vulkan1, 27.23
tok/s and 268.5 t/s prefill at 64k). It is not a regime change (+7 % decode),
and it costs the 3090 ~15 GiB — memory this fleet instead spends on the
vision endpoint, whose settled footprint is an Eras row on
[[frank-zealot]]. An operator who wants the 7 % must give up the fleet's only
image-capable endpoint.
2. **Per-tensor placement across two devices is impossible while `-ot` is used**,
for a scheduler-mechanical reason, regardless of driver or vendor mix.
The durable surprise worth keeping: **mainline llama.cpp + Vulkan on gfx1151 is a
viable engine for this class of model** (25.8 tok/s at 64k depth, and 27.6 with a
two-device split), which the 2026-09-15 mainline measurement of 1.01 tok/s did
*not* show — that number was the CPU+3090 hybrid, not the iGPU. Anyone re-opening
the "is there a better engine than halogen?" question should start here, with the
memory budget as the first constraint rather than the last.
[INFERENCE] Extrapolating A\*'s measured KV slope (32.64 → 25.77 from 0 to 64k
depth over 12 attention layers) to 200k gives roughly 20 tok/s, which is where
halogen's ~25 tok/s at real contexts wins outright — the deeper the context, the
worse mainline's position, and depth is filled by prefill that mainline would
have to spend ~200k/232 ≈ 14 minutes on.
## Reproducing
```bash
# dual-backend build (glslc + libvulkan are already installed on dankclaw)
cmake -B build-dual -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
-DGGML_CUDA=ON -DGGML_VULKAN=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CCACHE=ON
cmake --build build-dual --target llama-bench llama-cli llama-server -j 16
# device map on this host
CUDA0 = RTX 3090 Vulkan0 = RTX 3090 (same card)
Vulkan1 = Radeon Graphics (RADV STRIX_HALO) <- the iGPU, uma=1, 114 GiB GTT
export GGML_CUDA_NO_PINNED=1
# A* — iGPU only (the single-device baseline), depth-matched
llama-bench -m <gguf> -dev Vulkan1 -ngl 999 \
-ot per_layer_token_embd=CPU -n 32 -d 65536 -r 1
# G2 — pure-Vulkan two-device layer split (SLASH = within one configuration)
llama-bench -m <gguf> -dev Vulkan1/Vulkan0 -sm layer -ts 0.85/0.15 \
-ngl 999 -n 32 -d 65536 -r 1
# H — cross-vendor split: 3090 on CUDA, iGPU on Vulkan
llama-bench -m <gguf> -dev CUDA0/Vulkan1 -sm layer -ts 0.15/0.85 \
-ngl 999 -n 32 -d 65536 -r 1
# proof of placement — sample the 3090 while the run is loaded
nvidia-smi --query-gpu=memory.used --format=csv,noheader # expect 12–15 GiB
```
Operational notes for the next run:
- Mainline's pinning guard is `GGML_CUDA_NO_PINNED`; ik used
`GGML_CUDA_NO_PINNED_WEIGHTS`. Without it a CPU-overridden tensor can land in a
pinned `CUDA_Host` buffer — the 2026-09-15 108 GiB page-lock that starved this
host's co-tenants.
- `systemctl mask` is **refused** for a relinked root unit: frank-zealot's
`/etc/systemd/system/frank-zealot.service` is a symlink into the repo, so
`mask` exits with "File ... already exists". Stop it and rely on the absence
of a restart trigger, or move the symlink aside.
- Restarting frank-dragoon needs the quadlet source back in
`~/.config/containers/systemd/` plus `daemon-reload`; the generator re-creates
the unit from that file only. After the restore (and the host reboot, see the
landmines) its boot ledger read `400384 positions, ~27.2 GiB` against a 33.5
GiB budget — no clamp — with 12.3 GiB left for everything else (14.7 GiB
before the reboot: the same model, a more fragmented host afterwards — the
engine reported 240 free contiguous 2 MiB blocks against 475 earlier). That
ledger is an era reading of [[frank-dragoon]]; the block-census lesson it
teaches is on [[Llama Cpp Serving Sizing]]. Both endpoints answered `/health`
and served `/v1/models` under their names.
## Smaller-quant check: Q4_K_S against the recorded Q4_K_M numbers
Same host, same `build-dual`, same test shapes (`-r 1`, f16 KV, flash-attn
auto, `-d 65536` for depth). The repo ships i1-Q4_K_S as three `.partNofM`
fragments (35.00 + 35.00 + 34.09 GiB) to concatenate — 104.09 GiB against
110.97 for the Q4_K_M payload the earlier numbers came from.
| test | Q4_K_M | Q4_K_S | Δ |
|---|---|---|---|
| A\* iGPU-only pp512 / pp16384 / tg64 | 456.7 / 403.1 / 32.64 | **466.4 / 409.6 / 33.50** | +2.1 / +1.6 / +2.6 % |
| A\* iGPU-only pp512 / tg32 @64k | 231.8 / 25.77 | 232.0 / **26.47** | ~0 / +2.7 % |
| CUDA0/Vulkan1 split pp512 / tg32 @64k | 268.5 / 27.23 | 268.0 / **28.35** | ~0 / +4.1 % |
| Vulkan1/Vulkan0 split pp512 / tg32 @64k | 224.3 / 27.55 | **233.6 / 28.11** | +4.2 / +2.0 % |
Memory sampled through the run (`/proc/meminfo`, `mem_info_gtt_used` and
`nvidia-smi`, every 15 s): peak 3090 VRAM **13,466 MiB** (the CUDA-side split),
peak iGPU GTT **71.6 GiB** (A\*), and host RAM bottomed at **10.0 GiB
available** during A\* — against ~5 GiB for the Q4_K_M A\* run, so the 6.9 GiB
smaller file bought roughly that much headroom.
**Read it as a memory lever with a small speed dividend**: 2–4 % for 6.9 GiB.
Nothing structural moves — `-ot` still gates per-tensor placement and a layer
split still cannot consolidate KV on the 3090. If the aim were to *deploy*
mainline on this iGPU, `i1-IQ4_XS` (90.78 GiB, single file, sha256
`b4787265…d0574f9d`) is the rung that changes the memory arithmetic and q8_0 KV
is the lever that changes depth bandwidth; Q4_K_S is the conservative middle.
## The link, measured (2026-09-16)
`pcie-link-test.cu` — CUDA, pinned host buffers, `cudaEvent` timing, both
services stopped. Gen4 x4 spec ceiling is 7.877 GB/s usable; GB/s throughout.
| size | H2D sync | H2D async | D2H sync | 1× sync xfer | pageable H2D |
|---|---|---|---|---|---|
| 3 MiB | 6.65 | 6.71 | 6.54 | **474 µs** | 6.33 |
| 16 MiB | 6.72 | 6.72 | 6.59 | 2.50 ms | 6.61 |
| 64 MiB | 6.72 | 6.71 | 6.60 | 9.97 ms | 6.64 |
| 256 MiB | 6.73 | 6.73 | 6.60 | 39.9 ms | 6.67 |
| 1 GiB | 6.72 | 6.72 | 6.60 | 160 ms | 6.67 |
| 2 GiB | 6.72 | 6.72 | 6.60 | 320 ms | 6.67 |
Three readings:
- **6.72 GB/s H2D = 85 % of the Gen4 x4 ceiling** (D2H 6.60 = 84 %). The card is
x16-capable and the Strix Halo port beneath it is x8-capable
(`LnkCap: 16GT/s, Width x8` on `00:03.1`), so the x4 lives in the
adapter/cable, not the slot. The link idles at 2.5 GT/s x4 and trains up under
load — which these numbers prove on their own, since Gen1 x4 tops out near
1 GB/s.
- **Async buys nothing and pinning buys ~1 %**: one stream of copies already
saturates the link (6.71 async vs 6.65 sync at 3 MiB; identical at 1 GiB), and
pageable transfers run within noise of pinned ones. On this host the copies
are link-limited rather than staging-limited, so the usual "pageable costs 2×"
rule does not apply.
- **A 3 MiB slab costs 474 µs**, essentially `3 MiB / 6.72 GB/s` (447 µs) plus
~27 µs of launch. The streaming tax here is bandwidth, not per-copy overhead.
### What that means for an expert-streaming engine
Active expert bytes for this model are ~1.70 GiB/token (A3B @ Q4_K_M, 75 GiB
expert mass) — 580 × 3 MiB slabs:
| assumption | per-token link time | ceiling |
|---|---|---|
| every slab misses (bandwidth-bound) | 272 ms | **3.7 tok/s** |
| 20 % resident in a 24 GiB card's expert cache | 217 ms | 4.6 tok/s |
| 50 % hit rate (generous locality) | 136 ms | 7.4 tok/s |
| *the same bytes computed in host RAM at 256 GB/s* | 7.1 ms | the link is **31× slower** than the memory behind it |
Against this host's measured alternatives — **20.46 tok/s** for llama.cpp with
the experts *computed* on the CPU (link carries activations only, config B′
above), 25 for the halogen shape deployed that day ([[frank-dragoon]]), 26.5 for
mainline on the iGPU — an engine that *streams* experts across this link loses
by 3–5×, and the port's x8 ceiling would only double it (≈7–9 tok/s).
For the specific engine that raised the question: FreeToken v0.1.2 keeps expert
weights in host RAM with an LRU GPU cache — exactly this streaming shape — and
it loads only modelopt block-FP8 HF safetensors for `qwen3_5_moe`, so a Q4_K_M
**GGUF** is not loadable at all before bandwidth enters the discussion.