Frank
04f4e2c6d4
Enforces "one fact, one owner": technique pages keep mechanism, laws, traps and verdicts; per-endpoint shapes, digests and era measurements are cited to the endpoint page that owns them. Also repairs the four pointers the pass left empty (each verified against the cited owner before landing): - EMULATION dequant doubles host RAM (118 GiB -> ~238 GiB) -> [[frank-archon]] - 12.11 tok/s flat 304 -> 16,095 tokens -> [[Tcclaviger Fork Ops]] - bundled-head acceptance ladder 0.739/0.554/0.427/0.340 -> [[Qwen3.8 PP Deployment]] - llama.cpp PR #23897 dedup provenance -> [[qwen3.8-spec-matrix]] Replaces the retired-shape illustration in Llama Cpp Serving Sizing with the owned MiMo Q5_K_M row (6.40 GiB payload -> 20,530 MiB used), and unwraps a hard-wrapped [[Dankserver System Specs]] link that failed link integrity. Orphan census over the diff: 0 numbers lost (comma-insensitive match against the whole vault plus AGENTS.md).
233 lines
13 KiB
Markdown
233 lines
13 KiB
Markdown
---
|
||
type: reference
|
||
description: "This page answers one question: **how do we make Qwen3.8-27B serve faster on"
|
||
tags:
|
||
- llm
|
||
timestamp: 2026-09-03
|
||
---
|
||
|
||
# Qwen3.8-27B on Strix Halo: Speed Without Lowering the Quant
|
||
|
||
This page answers one question: **how do we make Qwen3.8-27B serve faster on
|
||
Dankclaw (Strix Halo, Radeon 8060S) while keeping UD-Q8_K_XL?** It records
|
||
the model's real architecture, the levers that are available, their evidence,
|
||
and the test matrix planned for 2026-08-16 morning.
|
||
|
||
Related pages:
|
||
- [Speculative-decoding matrix](qwen3.8-spec-matrix.md)
|
||
- [[frank-dragoon]] — the endpoint this host became; its Eras and Laws carry the
|
||
backend and spec-decode measurements.
|
||
- [Dankclaw Improvements](Dankclaw%20Improvements.md) (OS-level items)
|
||
- [Pipeline Parallelism: Qwen3.8 PP Deployment](Pipeline%20Parallelism/Qwen3.8%20PP%20Deployment.md)
|
||
|
||
## TL;DR — ranked levers
|
||
|
||
| # | Lever | Expected gain | Cost | Status |
|
||
|---|-------|--------------|------|--------|
|
||
| 1 | Pipeline-parallel to the RTX 3090 (RPC) | ~1.5–2× decode | 2.5 GbE link, both services stop for test | Kit ready, test 08-16 |
|
||
| 2 | MTP `n_max` explicit 4–5 + Q4_0 draft head | +5–15% on the spec share | restart, 1.56 GB file | test 08-16 |
|
||
| 3 | Plain `Q8_0` A/B vs UD-Q8_K_XL (same 8 bits) | dequant ALUs may differ | restart, 29 GB file | test 08-16 |
|
||
| 4 | Prefill path: ROCm backend + tuned ubatch/threads-batch | 2–3× prefill for agentic turns | per-workload backend choice | partial data |
|
||
| 5 | TTM `pages_limit` raised | removes the hard VRAM ceiling; the full 262k f16 KV plus the Q8 AGENT payload, MTP draft, and mmproj fit in one slot | modprobe, set live 2026-08-25 | applied — value and file: [[Dankclaw System Specs]] (Memory) |
|
||
| 6 | Vulkan ICD pinning + loader-layer disable | robustness; small | service env vars | test 08-16 |
|
||
|
||
Everything else (quantization downgrade, NVFP4, larger drafts) is documented
|
||
below for completeness but is out of scope per the "no lowering" constraint.
|
||
|
||
## What the model actually is
|
||
|
||
Verified from the GGUF header (`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL.gguf`,
|
||
dumped 2026-08-15) and the official model card.
|
||
|
||
- Architecture: `qwen35` / Qwen3.8-27B, **hybrid attention**.
|
||
- 64 main layers + 1 MTP (nextn) layer = 65 GGUF blocks.
|
||
- 48 Gated DeltaNet (linear/recurrent) layers + 16 gated full-attention
|
||
layers, in a 3:1 pattern (`full_attention_interval = 4`).
|
||
- Hidden 5120, FFN 17408, 24 Q heads / 4 KV heads, head dim 256.
|
||
- Native context 262144 (extendable per the card; production runs 262k/slot).
|
||
- GDN layers keep a **fixed-size** recurrent state (constant per layer, not
|
||
per token). Only the 16 full-attention layers grow with context.
|
||
- KV per token (full-attention part only): 16 layers × 2 (K,V) × 4 heads ×
|
||
256 dim × 1 byte (q8_0) = **32 KiB/token**. At 65536 ctx ≈ 2 GiB; at
|
||
262144 ctx ≈ 8 GiB.
|
||
- Sampling defaults (card, thinking mode on by default): temperature 1.0,
|
||
top-k 20, top-p 0.95, min-p 0.0. Production already matches.
|
||
|
||
Consequences of the hybrid design:
|
||
|
||
1. **Decode is cheap in memory.** Attention cost scales with only 16 layers;
|
||
the 48 GDN layers are O(1) per token. Long-context decode degrades far
|
||
slower than a dense transformer.
|
||
2. **Prompt cache reuse is unavailable upstream.** Recurrent/SSM state
|
||
cannot be prefix-cached the way attention KV can. Every agentic turn
|
||
re-prefills the whole conversation. **Prefill is the hidden cost of
|
||
agentic use**, and it is the dominant cost we have not yet optimized.
|
||
3. **MTP is native and strong.** The model ships a nextn head; on Vulkan at 16k
|
||
it roughly doubled the no-spec rate (the pair of figures is
|
||
[[qwen3.8-spec-matrix]]'s realistic-prompt table).
|
||
|
||
## Baseline this page optimized (2026-08-15)
|
||
|
||
`frank-dragoon` then ran llama.cpp on the iGPU with the UD-Q8_K_XL target
|
||
(bundled MTP head), `--spec-default` ngram-mod under a draft-mtp first link, two
|
||
262144 slots, q8_0 KV, batch 8192 / ubatch 256, threads 12. That era's ledger is
|
||
[[frank-dragoon]]'s; what this page keeps is the shape of the problem: the
|
||
service occupied **28 GiB of the then 96 GiB firmware VRAM carve, ~74 GiB
|
||
idle**, decode held ~25 t/s at 16k and collapsed at 262k depth
|
||
([[qwen3.8-spec-matrix]] §High-Context Backend Comparison), and prefill was the
|
||
unoptimized agentic cost (section 4).
|
||
|
||
## The levers, in detail
|
||
|
||
### 1. Pipeline-parallel to the RTX 3090 (biggest win)
|
||
|
||
The 3090 on Dankserver had headroom that night (it carried the day's small-model
|
||
service with a couple of GiB free — that card's eras are [[frank-zealot]]'s).
|
||
llama.cpp RPC splits the model by **layers**: each device runs its layer range,
|
||
hidden states cross the 2.5 GbE ethernet once per token. The per-position wire
|
||
bill is measured in [[Pipeline Parallelism/Ethernet Bandwidth Math]] §0; at
|
||
~0.5 ms LAN RTT the bandwidth cost is negligible — the split is latency-bound,
|
||
not bandwidth-bound.
|
||
|
||
Why this is a real speedup and not just "two GPUs": the 3090 does
|
||
matmuls roughly 2–3× faster per layer than the 8060S (936 vs 512 GB/s
|
||
memory, 80 vs 40 CUs). A 50/50 layer split balances per-layer latency, so
|
||
each token step takes ≈ max(half-on-3090, half-on-Strix) + one RTT instead
|
||
of all-64-layers-on-Strix.
|
||
|
||
Every decision this led to is written once, in
|
||
[[Pipeline Parallelism/Qwen3.8 PP Deployment]] §Gotchas (with the 4B smoke test
|
||
that proved each): head on the box that already holds the model file, **no
|
||
`--device` on the head** (setting it silently drops RPC), the fixed
|
||
HIP-before-Vulkan device selection, the reported-free-memory split trap, the
|
||
identical-build requirement, and `-dio` on the head — the last one visorcraft's
|
||
Strix finding, matching our own 28 GiB load-stall history. Scripts and the
|
||
morning runbook are on the same page.
|
||
|
||
### 2. Speculative-decoding tuning (same bits, free tokens)
|
||
|
||
- **MTP `n_max`.** Production left `--spec-draft-n-max` unset; the fleet matrix
|
||
(Vulkan, 16k, realistic prompt) found the depth-5 peak against the no-spec
|
||
baseline — both figures are [[qwen3.8-spec-matrix]]'s realistic-prompt table.
|
||
The published Strix tuning curve peaks at n_max 3–4 (30.3 t/s at 4 vs 28.9 at
|
||
3 vs 29.9 at 5). The PP env set **4** explicitly; the matrix re-tested 3/4/5
|
||
on the real split.
|
||
- **Q4_0 draft head.** The published Strix card measured a standalone
|
||
Q4_0 MTP head ~6–10% faster than Q8_0 for the same acceptance (the draft
|
||
head is a 248k-vocab output projection; cheaper dequant, same accuracy).
|
||
The bundled head inside UD-Q8_K_XL is Q8-class; the standalone
|
||
`ggml-org/mtp-Qwen3.8-27B-Q4_0.gguf` (1.56 GB) is being downloaded for
|
||
an A/B via `--spec-draft-model`.
|
||
- **Ngram-mod fallback stays.** It is the only spec type that survived parallel
|
||
load; the DSpark ROCm fault and the DFlash multi-slot corruption are timed and
|
||
explained on [[qwen3.8-spec-matrix]].
|
||
|
||
### 3. Same-bit quant A/B: UD-Q8_K_XL vs plain Q8_0
|
||
|
||
Both are 8-bit. UD (unsloth "dynamic") mixes block sizes per tensor; plain
|
||
`Q8_0` uses uniform 32-element blocks. Dequant of a smaller block costs
|
||
more ALUs per element; on a memory-bound GPU the difference is usually
|
||
small, but Strix's iGPU has tight ALU headroom. `ggml-org/Qwen3.8-27B-GGUF`
|
||
ships a Q8_0 (29.12 GB). One restart to test; if it wins, same quality at
|
||
zero quality cost.
|
||
|
||
### 4. Prefill is the agentic bottleneck — route it right
|
||
|
||
Every agentic turn re-prefills the whole conversation (no prefix caching for
|
||
hybrid state, upstream). Levers:
|
||
|
||
- **Backend choice per workload.** Measured @64k on this box (b10433, the era
|
||
[[frank-dragoon]] owns): ROCm 410 t/s vs Vulkan 302 t/s prefill; Vulkan 25.1
|
||
vs ROCm 24.3 t/s decode. For chat-heavy use: Vulkan. For agentic/batch-heavy
|
||
use: ROCm. The PP head (ROCm side) gets this for free.
|
||
- **ubatch/threads-batch.** Production runs ubatch 256. Larger ubatches
|
||
(512/1024) fill the iGPU better during prefill; `--threads-batch` for
|
||
the CPU-bound tail. Untested at prefill scale on this box — matrix item.
|
||
- **KV q4_0** for very long contexts: halves the 16-layer KV (8 GiB → 4 GiB
|
||
at 262k). Only worth it if long-ctx attention is the bottleneck; q8_0 is
|
||
the safe default.
|
||
|
||
### 5. OS level (from the Improvements doc, re-verified 2026-08-15)
|
||
|
||
- **TTM `pages_limit`** — the raise itself (value, live write, modprobe file,
|
||
date) is one row on [[Dankclaw System Specs]] (Memory). What matters to this
|
||
page is the mechanism: the driver-reported 96 GiB VRAM carve-out (VGM/SVM
|
||
pool) was never the ceiling — the TTM kernel page map was, and until it was
|
||
raised a 262k f16 KV cache plus a ~28 GB payload could not load in one slot.
|
||
- **Boot line** — IOMMU policy on this host is [[dankclaw-iommu-translation-off]]'s
|
||
and [[Dankclaw System Specs]]'s; it has changed twice since this lever was
|
||
written, so no era-scoped reading of `/proc/cmdline` belongs here.
|
||
- **Vulkan ICD pinning** — `radeon_icd.json` exists; no `VK_ICD_FILENAMES`
|
||
or loader-layer env is set. Pinning the radeon ICD and disabling loader
|
||
layers (`VK_LOADER_LAYERS_DISABLE=all`) is the recommended Strix
|
||
hardening; testable as service env vars, zero risk.
|
||
- **Power** — GPU DPM is already `performance`, forced level `auto`. The
|
||
`power_profile` sysfs file assumed in the Improvements doc does not exist
|
||
on this machine (checked 2026-08-15); that item is N/A here.
|
||
- Threads: 12 of 16 cores in production. Community reports 12–16 are
|
||
equivalent; leave at 12.
|
||
|
||
### 6. llama.cpp version watch
|
||
|
||
b10433 (2026-08-15) is adequate. Through b10451 there are no
|
||
spec-decoding fixes that help us; DFlash's multi-slot corruption is still
|
||
open upstream (fix #27117 pending). If that lands, DFlash/DSpark become
|
||
re-testable candidates (DSpark was the best single-machine draft before the
|
||
ROCm crash). No rebuild planned until then.
|
||
|
||
## Quantization landscape (reference)
|
||
|
||
The constraint is to keep UD-Q8_K_XL. This section records what exists so
|
||
future decisions are informed.
|
||
|
||
| Format | Size | Bits/weight | Notes |
|
||
|--------|------|-------------|-------|
|
||
| **UD-Q8_K_XL (unsloth)** | 31.5 GB | ~8.5 (mixed) | **Current.** Dynamic mixed-block; bundled MTP head; our baseline. |
|
||
| Q8_0 (ggml-org, bartowski) | 29.1 GB | 8.0 | Uniform 32-block. Same-bit A/B candidate (lever 3). |
|
||
| UD-Q6_K_XL (unsloth) | ~22 GB | ~6.5 | Below the 8-bit floor; documented, not recommended under the constraint. |
|
||
| Q6_K | ~21.5 GB | 6.6 | Same tier as above. |
|
||
| **ROCmFP4 "STRIX" (ROCmFPX fork)** | 13.3–13.8 GB | 4.25–4.39 | Strix-specific NVFP4-style format, ROCm backend only, requires a forked llama.cpp. Published: 30.3 t/s vs 10.7 for Q4_K_M (2.83×) at 8k with MTP n_max≈4; PPL ≈ Q4_K_M. **This is the "fast path" — and it is a quant downgrade plus a fork dependency that breaks the Vulkan production path.** Kept here as the escape hatch if the 8-bit speed target cannot be met. |
|
||
| NVFP4 (Blackwell) | ~14 GB | 4.5 | Hardware path for RTX 50-series. Irrelevant to 3090/Strix. |
|
||
|
||
If the 8-bit PP result (lever 1) is disappointing, the decision order is:
|
||
push PP split/latency tuning → ROCmFP4 as a *second* service (A/B, not a
|
||
replacement) → revisit the constraint with Frank.
|
||
|
||
## Morning test matrix (2026-08-16)
|
||
|
||
Policy: stop both production services, run tests, restore. Each config =
|
||
one head restart (~10 min load incl. weight stream; ~2 min with the worker
|
||
tensor cache warm).
|
||
|
||
| # | Config | Measures |
|
||
|---|--------|----------|
|
||
| A | PP split 1,1, ROCm head, MTP n_max=4 + ngram | baseline PP |
|
||
| B | A + `SPEC_NONE=1` | raw (no spec) PP speed |
|
||
| C | A but head build `build-vulkan-rpc` (Vulkan side) | backend A/B |
|
||
| D | A + `--spec-draft-model` Q4_0 MTP file | draft-head A/B |
|
||
| E | A with n_max 3 and 5 | n_max curve on the split |
|
||
| F | A, plain Q8_0 model file | same-bit A/B |
|
||
| G | PP default proportional split (no `--tensor-split`) | confirms the 20/80 handicap |
|
||
| Ref | production frank-dragoon (Vulkan, 2 slots) | the bar to beat |
|
||
|
||
Per config: `pp-test.sh bench 4` (4k prompt / 256 out) and `bench 16`
|
||
(16k / 256). Report: decode t/s (server print_timing), prefill t/s, MTP
|
||
acceptance, worker VRAM, head GTT usage.
|
||
|
||
Success bar: **≥ 30 t/s at 4k and ≥ 20 t/s at 16k decode**, against the
|
||
production @16k bar in the Baseline section, with the same 8-bit model.
|
||
|
||
## Sources
|
||
|
||
- GGUF header dump of the production model (this machine, 2026-08-15).
|
||
- Qwen3.8-27B model card + `ggml-org/Qwen3.8-27B-GGUF` /
|
||
`unsloth/Qwen3.8-27B-GGUF` repo listings (Hugging Face).
|
||
- kingjones `Qwen3.8-27B-ROCmFP4-GGUF` Strix tuning card (MTP n_max curve,
|
||
Q4_0 vs Q8_0 draft head, ROCmFP4 numbers).
|
||
- visorcraft `strix-halo-llm-perf` (Vulkan tg / ROCm prefill, ICD pinning,
|
||
`-dio` for RPC).
|
||
- llama.cpp b10433 source: `src/llama.cpp` (device selection),
|
||
`common/speculative.cpp` (MTP/ngram-mod), `ggml/src/ggml-rpc` (protocol).
|
||
- Local benchmarks: the measurement record lives on [[frank-dragoon]] (Eras and
|
||
Laws) and in [qwen3.8-spec-matrix](qwen3.8-spec-matrix.md).
|