Frank
a5fe045b58
Structural repair: - 13 pages were the same document twice (each exactly 2x its HEAD size), a second frontmatter block whose closing fence had been glued onto a body line by a frontmatter rewrite that re-read its own output. Un-doubled by cutting at the second frontmatter and keeping the copy whose body is a superset; the orphan census reports 0 lost tokens against HEAD. - log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first — 66 dated headings merged into 34, titles demoted to bold leads, every content line verified present. The file had been descending at the head and ascending at the tail. Ownership moves: - New `LLM/frank-medium.md` is the locator for the retired Dankserver endpoint: era windows quoted from each era page's own banner, no numbers restated, and the two unresolved date conflicts recorded instead of silently resolved. `frank-archon.md` links to it instead of restating the pre-rename window. - The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone; `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor verdicts and cross-split spec decode, and the four links that pointed at the stubs were repointed at it or at the drafter page. - Authentik's three stub pages fold into `Authentik.md`, with the process detail and the outpost reference split out so the parent stays under the length hint. Verified corrections (host census + live probe, 2026-10-19): - The MXFP4 checkpoint two pages called the "on-disk rollback" is gone: /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on the era page that owns that payload; the swap page now links to it. - frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2 arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run census is kept once (155 requests) instead of twice (~120 and 155). - Fleet shapes re-confirmed against the live endpoints before touching any page: 3x200000 fp8 KV, 2x200704, 2x262144. Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
233 lines
13 KiB
Markdown
233 lines
13 KiB
Markdown
---
|
||
type: reference
|
||
description: "How a served checkpoint got faster on the Strix Halo iGPU: the lever ladder and what each lever actually bought."
|
||
tags:
|
||
- llm
|
||
timestamp: 2026-09-03
|
||
---
|
||
|
||
# Qwen3.8-27B on Strix Halo: Speed Without Lowering the Quant
|
||
|
||
This page answers one question: **how do we make Qwen3.8-27B serve faster on
|
||
Dankclaw (Strix Halo, Radeon 8060S) while keeping UD-Q8_K_XL?** It records
|
||
the model's real architecture, the levers that are available, their evidence,
|
||
and the test matrix planned for 2026-08-16 morning.
|
||
|
||
Related pages:
|
||
- [Speculative-decoding matrix](qwen3.8-spec-matrix.md)
|
||
- [[frank-dragoon]] — the endpoint this host became; its Eras and Laws carry the
|
||
backend and spec-decode measurements.
|
||
- [Dankclaw Improvements](Dankclaw%20Improvements.md) (OS-level items)
|
||
- [Pipeline Parallelism: Qwen3.8 PP Deployment](Pipeline%20Parallelism/Qwen3.8%20PP%20Deployment.md)
|
||
|
||
## TL;DR — ranked levers
|
||
|
||
| # | Lever | Expected gain | Cost | Status |
|
||
|---|-------|--------------|------|--------|
|
||
| 1 | Pipeline-parallel to the RTX 3090 (RPC) | ~1.5–2× decode | 2.5 GbE link, both services stop for test | Kit ready, test 08-16 |
|
||
| 2 | MTP `n_max` explicit 4–5 + Q4_0 draft head | +5–15% on the spec share | restart, 1.56 GB file | test 08-16 |
|
||
| 3 | Plain `Q8_0` A/B vs UD-Q8_K_XL (same 8 bits) | dequant ALUs may differ | restart, 29 GB file | test 08-16 |
|
||
| 4 | Prefill path: ROCm backend + tuned ubatch/threads-batch | 2–3× prefill for agentic turns | per-workload backend choice | partial data |
|
||
| 5 | TTM `pages_limit` raised | removes the hard VRAM ceiling; the full 262k f16 KV plus the Q8 AGENT payload, MTP draft, and mmproj fit in one slot | modprobe, set live 2026-08-25 | applied — value and file: [[Dankclaw System Specs]] (Memory) |
|
||
| 6 | Vulkan ICD pinning + loader-layer disable | robustness; small | service env vars | test 08-16 |
|
||
|
||
Everything else (quantization downgrade, NVFP4, larger drafts) is documented
|
||
below for completeness but is out of scope per the "no lowering" constraint.
|
||
|
||
## What the model actually is
|
||
|
||
Verified from the GGUF header (`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL.gguf`,
|
||
dumped 2026-08-15) and the official model card.
|
||
|
||
- Architecture: `qwen35` / Qwen3.8-27B, **hybrid attention**.
|
||
- 64 main layers + 1 MTP (nextn) layer = 65 GGUF blocks.
|
||
- 48 Gated DeltaNet (linear/recurrent) layers + 16 gated full-attention
|
||
layers, in a 3:1 pattern (`full_attention_interval = 4`).
|
||
- Hidden 5120, FFN 17408, 24 Q heads / 4 KV heads, head dim 256.
|
||
- Native context 262144 (extendable per the card; production runs 262k/slot).
|
||
- GDN layers keep a **fixed-size** recurrent state (constant per layer, not
|
||
per token). Only the 16 full-attention layers grow with context.
|
||
- KV per token (full-attention part only): 16 layers × 2 (K,V) × 4 heads ×
|
||
256 dim × 1 byte (q8_0) = **32 KiB/token**. At 65536 ctx ≈ 2 GiB; at
|
||
262144 ctx ≈ 8 GiB.
|
||
- Sampling defaults (card, thinking mode on by default): temperature 1.0,
|
||
top-k 20, top-p 0.95, min-p 0.0. Production already matches.
|
||
|
||
Consequences of the hybrid design:
|
||
|
||
1. **Decode is cheap in memory.** Attention cost scales with only 16 layers;
|
||
the 48 GDN layers are O(1) per token. Long-context decode degrades far
|
||
slower than a dense transformer.
|
||
2. **Prompt cache reuse is unavailable upstream.** Recurrent/SSM state
|
||
cannot be prefix-cached the way attention KV can. Every agentic turn
|
||
re-prefills the whole conversation. **Prefill is the hidden cost of
|
||
agentic use**, and it is the dominant cost we have not yet optimized.
|
||
3. **MTP is native and strong.** The model ships a nextn head; on Vulkan at 16k
|
||
it roughly doubled the no-spec rate (the pair of figures is
|
||
[[qwen3.8-spec-matrix]]'s realistic-prompt table).
|
||
|
||
## Baseline this page optimized (2026-08-15)
|
||
|
||
`frank-dragoon` then ran llama.cpp on the iGPU with the UD-Q8_K_XL target
|
||
(bundled MTP head), `--spec-default` ngram-mod under a draft-mtp first link, two
|
||
262144 slots, q8_0 KV, batch 8192 / ubatch 256, threads 12. That era's ledger is
|
||
[[frank-dragoon]]'s; what this page keeps is the shape of the problem: the
|
||
service occupied **28 GiB of the then 96 GiB firmware VRAM carve, ~74 GiB
|
||
idle**, decode held ~25 t/s at 16k and collapsed at 262k depth
|
||
([[qwen3.8-spec-matrix]] §High-Context Backend Comparison), and prefill was the
|
||
unoptimized agentic cost (section 4).
|
||
|
||
## The levers, in detail
|
||
|
||
### 1. Pipeline-parallel to the RTX 3090 (biggest win)
|
||
|
||
The 3090 on Dankserver had headroom that night (it carried the day's small-model
|
||
service with a couple of GiB free — that card's eras are [[frank-zealot]]'s).
|
||
llama.cpp RPC splits the model by **layers**: each device runs its layer range,
|
||
hidden states cross the 2.5 GbE ethernet once per token. The per-position wire
|
||
bill is measured in [[Pipeline Parallelism/Ethernet Bandwidth Math]] §0; at
|
||
~0.5 ms LAN RTT the bandwidth cost is negligible — the split is latency-bound,
|
||
not bandwidth-bound.
|
||
|
||
Why this is a real speedup and not just "two GPUs": the 3090 does
|
||
matmuls roughly 2–3× faster per layer than the 8060S (936 vs 512 GB/s
|
||
memory, 80 vs 40 CUs). A 50/50 layer split balances per-layer latency, so
|
||
each token step takes ≈ max(half-on-3090, half-on-Strix) + one RTT instead
|
||
of all-64-layers-on-Strix.
|
||
|
||
Every decision this led to is written once, in
|
||
[[Pipeline Parallelism/Qwen3.8 PP Deployment]] §Gotchas (with the 4B smoke test
|
||
that proved each): head on the box that already holds the model file, **no
|
||
`--device` on the head** (setting it silently drops RPC), the fixed
|
||
HIP-before-Vulkan device selection, the reported-free-memory split trap, the
|
||
identical-build requirement, and `-dio` on the head — the last one visorcraft's
|
||
Strix finding, matching our own 28 GiB load-stall history. Scripts and the
|
||
morning runbook are on the same page.
|
||
|
||
### 2. Speculative-decoding tuning (same bits, free tokens)
|
||
|
||
- **MTP `n_max`.** Production left `--spec-draft-n-max` unset; the fleet matrix
|
||
(Vulkan, 16k, realistic prompt) found the depth-5 peak against the no-spec
|
||
baseline — both figures are [[qwen3.8-spec-matrix]]'s realistic-prompt table.
|
||
The published Strix tuning curve peaks at n_max 3–4 (30.3 t/s at 4 vs 28.9 at
|
||
3 vs 29.9 at 5). The PP env set **4** explicitly; the matrix re-tested 3/4/5
|
||
on the real split.
|
||
- **Q4_0 draft head.** The published Strix card measured a standalone
|
||
Q4_0 MTP head ~6–10% faster than Q8_0 for the same acceptance (the draft
|
||
head is a 248k-vocab output projection; cheaper dequant, same accuracy).
|
||
The bundled head inside UD-Q8_K_XL is Q8-class; the standalone
|
||
`ggml-org/mtp-Qwen3.8-27B-Q4_0.gguf` (1.56 GB) is being downloaded for
|
||
an A/B via `--spec-draft-model`.
|
||
- **Ngram-mod fallback stays.** It is the only spec type that survived parallel
|
||
load; the DSpark ROCm fault and the DFlash multi-slot corruption are timed and
|
||
explained on [[qwen3.8-spec-matrix]].
|
||
|
||
### 3. Same-bit quant A/B: UD-Q8_K_XL vs plain Q8_0
|
||
|
||
Both are 8-bit. UD (unsloth "dynamic") mixes block sizes per tensor; plain
|
||
`Q8_0` uses uniform 32-element blocks. Dequant of a smaller block costs
|
||
more ALUs per element; on a memory-bound GPU the difference is usually
|
||
small, but Strix's iGPU has tight ALU headroom. `ggml-org/Qwen3.8-27B-GGUF`
|
||
ships a Q8_0 (29.12 GB). One restart to test; if it wins, same quality at
|
||
zero quality cost.
|
||
|
||
### 4. Prefill is the agentic bottleneck — route it right
|
||
|
||
Every agentic turn re-prefills the whole conversation (no prefix caching for
|
||
hybrid state, upstream). Levers:
|
||
|
||
- **Backend choice per workload.** Measured @64k on this box (b10433, the era
|
||
[[frank-dragoon]] owns): ROCm 410 t/s vs Vulkan 302 t/s prefill; Vulkan 25.1
|
||
vs ROCm 24.3 t/s decode. For chat-heavy use: Vulkan. For agentic/batch-heavy
|
||
use: ROCm. The PP head (ROCm side) gets this for free.
|
||
- **ubatch/threads-batch.** Production runs ubatch 256. Larger ubatches
|
||
(512/1024) fill the iGPU better during prefill; `--threads-batch` for
|
||
the CPU-bound tail. Untested at prefill scale on this box — matrix item.
|
||
- **KV q4_0** for very long contexts: halves the 16-layer KV (8 GiB → 4 GiB
|
||
at 262k). Only worth it if long-ctx attention is the bottleneck; q8_0 is
|
||
the safe default.
|
||
|
||
### 5. OS level (from the Improvements doc, re-verified 2026-08-15)
|
||
|
||
- **TTM `pages_limit`** — the raise itself (value, live write, modprobe file,
|
||
date) is one row on [[Dankclaw System Specs]] (Memory). What matters to this
|
||
page is the mechanism: the driver-reported 96 GiB VRAM carve-out (VGM/SVM
|
||
pool) was never the ceiling — the TTM kernel page map was, and until it was
|
||
raised a 262k f16 KV cache plus a ~28 GB payload could not load in one slot.
|
||
- **Boot line** — IOMMU policy on this host is [[dankclaw-iommu-translation-off]]'s
|
||
and [[Dankclaw System Specs]]'s; it has changed twice since this lever was
|
||
written, so no era-scoped reading of `/proc/cmdline` belongs here.
|
||
- **Vulkan ICD pinning** — `radeon_icd.json` exists; no `VK_ICD_FILENAMES`
|
||
or loader-layer env is set. Pinning the radeon ICD and disabling loader
|
||
layers (`VK_LOADER_LAYERS_DISABLE=all`) is the recommended Strix
|
||
hardening; testable as service env vars, zero risk.
|
||
- **Power** — GPU DPM is already `performance`, forced level `auto`. The
|
||
`power_profile` sysfs file assumed in the Improvements doc does not exist
|
||
on this machine (checked 2026-08-15); that item is N/A here.
|
||
- Threads: 12 of 16 cores in production. Community reports 12–16 are
|
||
equivalent; leave at 12.
|
||
|
||
### 6. llama.cpp version watch
|
||
|
||
b10433 (2026-08-15) is adequate. Through b10451 there are no
|
||
spec-decoding fixes that help us; DFlash's multi-slot corruption is still
|
||
open upstream (fix #27117 pending). If that lands, DFlash/DSpark become
|
||
re-testable candidates (DSpark was the best single-machine draft before the
|
||
ROCm crash). No rebuild planned until then.
|
||
|
||
## Quantization landscape (reference)
|
||
|
||
The constraint is to keep UD-Q8_K_XL. This section records what exists so
|
||
future decisions are informed.
|
||
|
||
| Format | Size | Bits/weight | Notes |
|
||
|--------|------|-------------|-------|
|
||
| **UD-Q8_K_XL (unsloth)** | 31.5 GB | ~8.5 (mixed) | **Current.** Dynamic mixed-block; bundled MTP head; our baseline. |
|
||
| Q8_0 (ggml-org, bartowski) | 29.1 GB | 8.0 | Uniform 32-block. Same-bit A/B candidate (lever 3). |
|
||
| UD-Q6_K_XL (unsloth) | ~22 GB | ~6.5 | Below the 8-bit floor; documented, not recommended under the constraint. |
|
||
| Q6_K | ~21.5 GB | 6.6 | Same tier as above. |
|
||
| **ROCmFP4 "STRIX" (ROCmFPX fork)** | 13.3–13.8 GB | 4.25–4.39 | Strix-specific NVFP4-style format, ROCm backend only, requires a forked llama.cpp. Published: 30.3 t/s vs 10.7 for Q4_K_M (2.83×) at 8k with MTP n_max≈4; PPL ≈ Q4_K_M. **This is the "fast path" — and it is a quant downgrade plus a fork dependency that breaks the Vulkan production path.** Kept here as the escape hatch if the 8-bit speed target cannot be met. |
|
||
| NVFP4 (Blackwell) | ~14 GB | 4.5 | Hardware path for RTX 50-series. Irrelevant to 3090/Strix. |
|
||
|
||
If the 8-bit PP result (lever 1) is disappointing, the decision order is:
|
||
push PP split/latency tuning → ROCmFP4 as a *second* service (A/B, not a
|
||
replacement) → revisit the constraint with Frank.
|
||
|
||
## Morning test matrix (2026-08-16)
|
||
|
||
Policy: stop both production services, run tests, restore. Each config =
|
||
one head restart (~10 min load incl. weight stream; ~2 min with the worker
|
||
tensor cache warm).
|
||
|
||
| # | Config | Measures |
|
||
|---|--------|----------|
|
||
| A | PP split 1,1, ROCm head, MTP n_max=4 + ngram | baseline PP |
|
||
| B | A + `SPEC_NONE=1` | raw (no spec) PP speed |
|
||
| C | A but head build `build-vulkan-rpc` (Vulkan side) | backend A/B |
|
||
| D | A + `--spec-draft-model` Q4_0 MTP file | draft-head A/B |
|
||
| E | A with n_max 3 and 5 | n_max curve on the split |
|
||
| F | A, plain Q8_0 model file | same-bit A/B |
|
||
| G | PP default proportional split (no `--tensor-split`) | confirms the 20/80 handicap |
|
||
| Ref | production frank-dragoon (Vulkan, 2 slots) | the bar to beat |
|
||
|
||
Per config: `pp-test.sh bench 4` (4k prompt / 256 out) and `bench 16`
|
||
(16k / 256). Report: decode t/s (server print_timing), prefill t/s, MTP
|
||
acceptance, worker VRAM, head GTT usage.
|
||
|
||
Success bar: **≥ 30 t/s at 4k and ≥ 20 t/s at 16k decode**, against the
|
||
production @16k bar in the Baseline section, with the same 8-bit model.
|
||
|
||
## Sources
|
||
|
||
- GGUF header dump of the production model (this machine, 2026-08-15).
|
||
- Qwen3.8-27B model card + `ggml-org/Qwen3.8-27B-GGUF` /
|
||
`unsloth/Qwen3.8-27B-GGUF` repo listings (Hugging Face).
|
||
- kingjones `Qwen3.8-27B-ROCmFP4-GGUF` Strix tuning card (MTP n_max curve,
|
||
Q4_0 vs Q8_0 draft head, ROCmFP4 numbers).
|
||
- visorcraft `strix-halo-llm-perf` (Vulkan tg / ROCm prefill, ICD pinning,
|
||
`-dio` for RPC).
|
||
- llama.cpp b10433 source: `src/llama.cpp` (device selection),
|
||
`common/speculative.cpp` (MTP/ngram-mod), `ggml/src/ggml-rpc` (protocol).
|
||
- Local benchmarks: the measurement record lives on [[frank-dragoon]] (Eras and
|
||
Laws) and in [qwen3.8-spec-matrix](qwen3.8-spec-matrix.md).
|