--- type: reference description: "This page answers one question: **how do we make Qwen3.8-27B serve faster on" tags: - llm timestamp: 2026-09-03 --- # Qwen3.8-27B on Strix Halo: Speed Without Lowering the Quant This page answers one question: **how do we make Qwen3.8-27B serve faster on Dankclaw (Strix Halo, Radeon 8060S) while keeping UD-Q8_K_XL?** It records the model's real architecture, the levers that are available, their evidence, and the test matrix planned for 2026-08-16 morning. Related pages: - [Speculative-decoding matrix](qwen3.8-spec-matrix.md) - [[frank-dragoon]] — the endpoint this host became; its Eras and Laws carry the backend and spec-decode measurements. - [Dankclaw Improvements](Dankclaw%20Improvements.md) (OS-level items) - [Pipeline Parallelism: Qwen3.8 PP Deployment](Pipeline%20Parallelism/Qwen3.8%20PP%20Deployment.md) ## TL;DR — ranked levers | # | Lever | Expected gain | Cost | Status | |---|-------|--------------|------|--------| | 1 | Pipeline-parallel to the RTX 3090 (RPC) | ~1.5–2× decode | 2.5 GbE link, both services stop for test | Kit ready, test 08-16 | | 2 | MTP `n_max` explicit 4–5 + Q4_0 draft head | +5–15% on the spec share | restart, 1.56 GB file | test 08-16 | | 3 | Plain `Q8_0` A/B vs UD-Q8_K_XL (same 8 bits) | dequant ALUs may differ | restart, 29 GB file | test 08-16 | | 4 | Prefill path: ROCm backend + tuned ubatch/threads-batch | 2–3× prefill for agentic turns | per-workload backend choice | partial data | | 5 | TTM `pages_limit` raised | removes the hard VRAM ceiling; the full 262k f16 KV plus the Q8 AGENT payload, MTP draft, and mmproj fit in one slot | modprobe, set live 2026-08-25 | applied — value and file: [[Dankclaw System Specs]] (Memory) | | 6 | Vulkan ICD pinning + loader-layer disable | robustness; small | service env vars | test 08-16 | Everything else (quantization downgrade, NVFP4, larger drafts) is documented below for completeness but is out of scope per the "no lowering" constraint. ## What the model actually is Verified from the GGUF header (`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL.gguf`, dumped 2026-08-15) and the official model card. - Architecture: `qwen35` / Qwen3.8-27B, **hybrid attention**. - 64 main layers + 1 MTP (nextn) layer = 65 GGUF blocks. - 48 Gated DeltaNet (linear/recurrent) layers + 16 gated full-attention layers, in a 3:1 pattern (`full_attention_interval = 4`). - Hidden 5120, FFN 17408, 24 Q heads / 4 KV heads, head dim 256. - Native context 262144 (extendable per the card; production runs 262k/slot). - GDN layers keep a **fixed-size** recurrent state (constant per layer, not per token). Only the 16 full-attention layers grow with context. - KV per token (full-attention part only): 16 layers × 2 (K,V) × 4 heads × 256 dim × 1 byte (q8_0) = **32 KiB/token**. At 65536 ctx ≈ 2 GiB; at 262144 ctx ≈ 8 GiB. - Sampling defaults (card, thinking mode on by default): temperature 1.0, top-k 20, top-p 0.95, min-p 0.0. Production already matches. Consequences of the hybrid design: 1. **Decode is cheap in memory.** Attention cost scales with only 16 layers; the 48 GDN layers are O(1) per token. Long-context decode degrades far slower than a dense transformer. 2. **Prompt cache reuse is unavailable upstream.** Recurrent/SSM state cannot be prefix-cached the way attention KV can. Every agentic turn re-prefills the whole conversation. **Prefill is the hidden cost of agentic use**, and it is the dominant cost we have not yet optimized. 3. **MTP is native and strong.** The model ships a nextn head; on Vulkan at 16k it roughly doubled the no-spec rate (the pair of figures is [[qwen3.8-spec-matrix]]'s realistic-prompt table). ## Baseline this page optimized (2026-08-15) `frank-dragoon` then ran llama.cpp on the iGPU with the UD-Q8_K_XL target (bundled MTP head), `--spec-default` ngram-mod under a draft-mtp first link, two 262144 slots, q8_0 KV, batch 8192 / ubatch 256, threads 12. That era's ledger is [[frank-dragoon]]'s; what this page keeps is the shape of the problem: the service occupied **28 GiB of the then 96 GiB firmware VRAM carve, ~74 GiB idle**, decode held ~25 t/s at 16k and collapsed at 262k depth ([[qwen3.8-spec-matrix]] §High-Context Backend Comparison), and prefill was the unoptimized agentic cost (section 4). ## The levers, in detail ### 1. Pipeline-parallel to the RTX 3090 (biggest win) The 3090 on Dankserver had headroom that night (it carried the day's small-model service with a couple of GiB free — that card's eras are [[frank-zealot]]'s). llama.cpp RPC splits the model by **layers**: each device runs its layer range, hidden states cross the 2.5 GbE ethernet once per token. The per-position wire bill is measured in [[Pipeline Parallelism/Ethernet Bandwidth Math]] §0; at ~0.5 ms LAN RTT the bandwidth cost is negligible — the split is latency-bound, not bandwidth-bound. Why this is a real speedup and not just "two GPUs": the 3090 does matmuls roughly 2–3× faster per layer than the 8060S (936 vs 512 GB/s memory, 80 vs 40 CUs). A 50/50 layer split balances per-layer latency, so each token step takes ≈ max(half-on-3090, half-on-Strix) + one RTT instead of all-64-layers-on-Strix. Every decision this led to is written once, in [[Pipeline Parallelism/Qwen3.8 PP Deployment]] §Gotchas (with the 4B smoke test that proved each): head on the box that already holds the model file, **no `--device` on the head** (setting it silently drops RPC), the fixed HIP-before-Vulkan device selection, the reported-free-memory split trap, the identical-build requirement, and `-dio` on the head — the last one visorcraft's Strix finding, matching our own 28 GiB load-stall history. Scripts and the morning runbook are on the same page. ### 2. Speculative-decoding tuning (same bits, free tokens) - **MTP `n_max`.** Production left `--spec-draft-n-max` unset; the fleet matrix (Vulkan, 16k, realistic prompt) found the depth-5 peak against the no-spec baseline — both figures are [[qwen3.8-spec-matrix]]'s realistic-prompt table. The published Strix tuning curve peaks at n_max 3–4 (30.3 t/s at 4 vs 28.9 at 3 vs 29.9 at 5). The PP env set **4** explicitly; the matrix re-tested 3/4/5 on the real split. - **Q4_0 draft head.** The published Strix card measured a standalone Q4_0 MTP head ~6–10% faster than Q8_0 for the same acceptance (the draft head is a 248k-vocab output projection; cheaper dequant, same accuracy). The bundled head inside UD-Q8_K_XL is Q8-class; the standalone `ggml-org/mtp-Qwen3.8-27B-Q4_0.gguf` (1.56 GB) is being downloaded for an A/B via `--spec-draft-model`. - **Ngram-mod fallback stays.** It is the only spec type that survived parallel load; the DSpark ROCm fault and the DFlash multi-slot corruption are timed and explained on [[qwen3.8-spec-matrix]]. ### 3. Same-bit quant A/B: UD-Q8_K_XL vs plain Q8_0 Both are 8-bit. UD (unsloth "dynamic") mixes block sizes per tensor; plain `Q8_0` uses uniform 32-element blocks. Dequant of a smaller block costs more ALUs per element; on a memory-bound GPU the difference is usually small, but Strix's iGPU has tight ALU headroom. `ggml-org/Qwen3.8-27B-GGUF` ships a Q8_0 (29.12 GB). One restart to test; if it wins, same quality at zero quality cost. ### 4. Prefill is the agentic bottleneck — route it right Every agentic turn re-prefills the whole conversation (no prefix caching for hybrid state, upstream). Levers: - **Backend choice per workload.** Measured @64k on this box (b10433, the era [[frank-dragoon]] owns): ROCm 410 t/s vs Vulkan 302 t/s prefill; Vulkan 25.1 vs ROCm 24.3 t/s decode. For chat-heavy use: Vulkan. For agentic/batch-heavy use: ROCm. The PP head (ROCm side) gets this for free. - **ubatch/threads-batch.** Production runs ubatch 256. Larger ubatches (512/1024) fill the iGPU better during prefill; `--threads-batch` for the CPU-bound tail. Untested at prefill scale on this box — matrix item. - **KV q4_0** for very long contexts: halves the 16-layer KV (8 GiB → 4 GiB at 262k). Only worth it if long-ctx attention is the bottleneck; q8_0 is the safe default. ### 5. OS level (from the Improvements doc, re-verified 2026-08-15) - **TTM `pages_limit`** — the raise itself (value, live write, modprobe file, date) is one row on [[Dankclaw System Specs]] (Memory). What matters to this page is the mechanism: the driver-reported 96 GiB VRAM carve-out (VGM/SVM pool) was never the ceiling — the TTM kernel page map was, and until it was raised a 262k f16 KV cache plus a ~28 GB payload could not load in one slot. - **Boot line** — IOMMU policy on this host is [[dankclaw-iommu-translation-off]]'s and [[Dankclaw System Specs]]'s; it has changed twice since this lever was written, so no era-scoped reading of `/proc/cmdline` belongs here. - **Vulkan ICD pinning** — `radeon_icd.json` exists; no `VK_ICD_FILENAMES` or loader-layer env is set. Pinning the radeon ICD and disabling loader layers (`VK_LOADER_LAYERS_DISABLE=all`) is the recommended Strix hardening; testable as service env vars, zero risk. - **Power** — GPU DPM is already `performance`, forced level `auto`. The `power_profile` sysfs file assumed in the Improvements doc does not exist on this machine (checked 2026-08-15); that item is N/A here. - Threads: 12 of 16 cores in production. Community reports 12–16 are equivalent; leave at 12. ### 6. llama.cpp version watch b10433 (2026-08-15) is adequate. Through b10451 there are no spec-decoding fixes that help us; DFlash's multi-slot corruption is still open upstream (fix #27117 pending). If that lands, DFlash/DSpark become re-testable candidates (DSpark was the best single-machine draft before the ROCm crash). No rebuild planned until then. ## Quantization landscape (reference) The constraint is to keep UD-Q8_K_XL. This section records what exists so future decisions are informed. | Format | Size | Bits/weight | Notes | |--------|------|-------------|-------| | **UD-Q8_K_XL (unsloth)** | 31.5 GB | ~8.5 (mixed) | **Current.** Dynamic mixed-block; bundled MTP head; our baseline. | | Q8_0 (ggml-org, bartowski) | 29.1 GB | 8.0 | Uniform 32-block. Same-bit A/B candidate (lever 3). | | UD-Q6_K_XL (unsloth) | ~22 GB | ~6.5 | Below the 8-bit floor; documented, not recommended under the constraint. | | Q6_K | ~21.5 GB | 6.6 | Same tier as above. | | **ROCmFP4 "STRIX" (ROCmFPX fork)** | 13.3–13.8 GB | 4.25–4.39 | Strix-specific NVFP4-style format, ROCm backend only, requires a forked llama.cpp. Published: 30.3 t/s vs 10.7 for Q4_K_M (2.83×) at 8k with MTP n_max≈4; PPL ≈ Q4_K_M. **This is the "fast path" — and it is a quant downgrade plus a fork dependency that breaks the Vulkan production path.** Kept here as the escape hatch if the 8-bit speed target cannot be met. | | NVFP4 (Blackwell) | ~14 GB | 4.5 | Hardware path for RTX 50-series. Irrelevant to 3090/Strix. | If the 8-bit PP result (lever 1) is disappointing, the decision order is: push PP split/latency tuning → ROCmFP4 as a *second* service (A/B, not a replacement) → revisit the constraint with Frank. ## Morning test matrix (2026-08-16) Policy: stop both production services, run tests, restore. Each config = one head restart (~10 min load incl. weight stream; ~2 min with the worker tensor cache warm). | # | Config | Measures | |---|--------|----------| | A | PP split 1,1, ROCm head, MTP n_max=4 + ngram | baseline PP | | B | A + `SPEC_NONE=1` | raw (no spec) PP speed | | C | A but head build `build-vulkan-rpc` (Vulkan side) | backend A/B | | D | A + `--spec-draft-model` Q4_0 MTP file | draft-head A/B | | E | A with n_max 3 and 5 | n_max curve on the split | | F | A, plain Q8_0 model file | same-bit A/B | | G | PP default proportional split (no `--tensor-split`) | confirms the 20/80 handicap | | Ref | production frank-dragoon (Vulkan, 2 slots) | the bar to beat | Per config: `pp-test.sh bench 4` (4k prompt / 256 out) and `bench 16` (16k / 256). Report: decode t/s (server print_timing), prefill t/s, MTP acceptance, worker VRAM, head GTT usage. Success bar: **≥ 30 t/s at 4k and ≥ 20 t/s at 16k decode**, against the production @16k bar in the Baseline section, with the same 8-bit model. ## Sources - GGUF header dump of the production model (this machine, 2026-08-15). - Qwen3.8-27B model card + `ggml-org/Qwen3.8-27B-GGUF` / `unsloth/Qwen3.8-27B-GGUF` repo listings (Hugging Face). - kingjones `Qwen3.8-27B-ROCmFP4-GGUF` Strix tuning card (MTP n_max curve, Q4_0 vs Q8_0 draft head, ROCmFP4 numbers). - visorcraft `strix-halo-llm-perf` (Vulkan tg / ROCm prefill, ICD pinning, `-dio` for RPC). - llama.cpp b10433 source: `src/llama.cpp` (device selection), `common/speculative.cpp` (MTP/ngram-mod), `ggml/src/ggml-rpc` (protocol). - Local benchmarks: the measurement record lives on [[frank-dragoon]] (Eras and Laws) and in [qwen3.8-spec-matrix](qwen3.8-spec-matrix.md).