Frank
017d3704d2
The LLM fleet documentation (deployment runbook, model runbooks, benchmarks, research archive, retired-topology history) joins the project knowledge base as docs/LLM/. Wikilinks resolve by filename, so no link updates are needed for the move.
248 lines
13 KiB
Markdown
248 lines
13 KiB
Markdown
# Qwen3.8-27B on Strix Halo: Speed Without Lowering the Quant
|
||
|
||
This page answers one question: **how do we make Qwen3.8-27B serve faster on
|
||
Dankclaw (Strix Halo, Radeon 8060S) while keeping UD-Q8_K_XL?** It records
|
||
the model's real architecture, the levers that are available, their evidence,
|
||
and the test matrix planned for 2026-08-16 morning.
|
||
|
||
Related pages:
|
||
- [Speculative-decoding matrix](qwen3.8-spec-matrix.md)
|
||
- [Backend and spec benchmark](frank-default-large-benchmark.md)
|
||
- [Dankclaw Improvements](Dankclaw%20Improvements.md) (OS-level items)
|
||
- [Pipeline Parallelism: Qwen3.8 PP Deployment](Pipeline%20Parallelism/Qwen3.8%20PP%20Deployment.md)
|
||
|
||
## TL;DR — ranked levers
|
||
|
||
| # | Lever | Expected gain | Cost | Status |
|
||
|---|-------|--------------|------|--------|
|
||
| 1 | Pipeline-parallel to the RTX 3090 (RPC) | ~1.5–2× decode | 2.5 GbE link, both services stop for test | Kit ready, test 08-16 |
|
||
| 2 | MTP `n_max` explicit 4–5 + Q4_0 draft head | +5–15% on the spec share | restart, 1.56 GB file | test 08-16 |
|
||
| 3 | Plain `Q8_0` A/B vs UD-Q8_K_XL (same 8 bits) | dequant ALUs may differ | restart, 29 GB file | test 08-16 |
|
||
| 4 | Prefill path: ROCm backend + tuned ubatch/threads-batch | 2–3× prefill for agentic turns | per-workload backend choice | partial data |
|
||
| 5 | TTM `pages_limit` raise (15.2 GiB → 90+) | removes a hard VRAM ceiling, stable 96 GiB budget | reboot | pending |
|
||
| 6 | Vulkan ICD pinning + loader-layer disable | robustness; small | service env vars | test 08-16 |
|
||
|
||
Everything else (quantization downgrade, NVFP4, larger drafts) is documented
|
||
below for completeness but is out of scope per the "no lowering" constraint.
|
||
|
||
## What the model actually is
|
||
|
||
Verified from the GGUF header (`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL.gguf`,
|
||
dumped 2026-08-15) and the official model card.
|
||
|
||
- Architecture: `qwen35` / Qwen3.8-27B, **hybrid attention**.
|
||
- 64 main layers + 1 MTP (nextn) layer = 65 GGUF blocks.
|
||
- 48 Gated DeltaNet (linear/recurrent) layers + 16 gated full-attention
|
||
layers, in a 3:1 pattern (`full_attention_interval = 4`).
|
||
- Hidden 5120, FFN 17408, 24 Q heads / 4 KV heads, head dim 256.
|
||
- Native context 262144 (extendable per the card; production runs 262k/slot).
|
||
- GDN layers keep a **fixed-size** recurrent state (constant per layer, not
|
||
per token). Only the 16 full-attention layers grow with context.
|
||
- KV per token (full-attention part only): 16 layers × 2 (K,V) × 4 heads ×
|
||
256 dim × 1 byte (q8_0) = **32 KiB/token**. At 65536 ctx ≈ 2 GiB; at
|
||
262144 ctx ≈ 8 GiB.
|
||
- Sampling defaults (card, thinking mode on by default): temperature 1.0,
|
||
top-k 20, top-p 0.95, min-p 0.0. Production already matches.
|
||
|
||
Consequences of the hybrid design:
|
||
|
||
1. **Decode is cheap in memory.** Attention cost scales with only 16 layers;
|
||
the 48 GDN layers are O(1) per token. Long-context decode degrades far
|
||
slower than a dense transformer.
|
||
2. **Prompt cache reuse is unavailable upstream.** Recurrent/SSM state
|
||
cannot be prefix-cached the way attention KV can. Every agentic turn
|
||
re-prefills the whole conversation. **Prefill is the hidden cost of
|
||
agentic use**, and it is the dominant cost we have not yet optimized.
|
||
3. **MTP is native and strong.** The model ships a nextn head; our matrix
|
||
measured 13.3 t/s vs 7.2 t/s (no spec) at 16k on Vulkan.
|
||
|
||
## Current production state (baseline)
|
||
|
||
`frank-default-large` (Dankclaw, Vulkan0, 2026-08-15):
|
||
|
||
- Model: UD-Q8_K_XL, 31.5 GB, bundled MTP head.
|
||
- `--spec-default` (ngram-mod) with draft-mtp first in the chain.
|
||
- 2 parallel slots × 262144 ctx, KV q8_0, batch 8192, ubatch 256,
|
||
threads 12, `-dio` off, `--fit off`.
|
||
- Measured: ~25 t/s at 16k ctx; 5.68 t/s at 262k ctx; prefill 302 t/s @64k
|
||
(Vulkan) vs 410 t/s @64k (ROCm, same build).
|
||
- GPU: 28 GiB of 96 GiB GTT used. **74 GiB sits idle.**
|
||
|
||
## The levers, in detail
|
||
|
||
### 1. Pipeline-parallel to the RTX 3090 (biggest win)
|
||
|
||
The 3090 on Dankserver is idle (serves a 22 GB MoE small-model service with
|
||
~2.4 GB free). llama.cpp RPC splits the model by **layers**: each device
|
||
runs its layer range, hidden states cross the 2.5 GbE ethernet once per
|
||
token. The per-token transfer is ~41 KiB (5120-wide f32 hidden state, both
|
||
directions); at ~0.5 ms LAN RTT the bandwidth cost is negligible — the split
|
||
is latency-bound, not bandwidth-bound (see
|
||
[Ethernet Bandwidth Math](Pipeline%20Parallelism/Ethernet%20Bandwidth%20Math.md)).
|
||
|
||
Why this is a real speedup and not just "two GPUs": the 3090 does
|
||
matmuls roughly 2–3× faster per layer than the 8060S (936 vs 512 GB/s
|
||
memory, 80 vs 40 CUs). A 50/50 layer split balances per-layer latency, so
|
||
each token step takes ≈ max(half-on-3090, half-on-Strix) + one RTT instead
|
||
of all-64-layers-on-Strix.
|
||
|
||
Design decisions (verified in b10433 source, 2026-08-15):
|
||
|
||
- **Head on Dankclaw, worker on Dankserver.** The 31.5 GB model file stays
|
||
on the head; the worker receives its tensor share over the wire at load
|
||
(with `-c` it caches the tensors locally after the first load).
|
||
- **Head must NOT pass `--device`.** When `params.devices` is set,
|
||
`llama_prepare_model_devices` builds the device list from exactly those
|
||
devices and the RPC device is silently dropped (`--rpc` becomes a no-op).
|
||
Default device selection inserts RPC servers **at the front** of the list.
|
||
- **The local iGPU side runs on ROCm, not Vulkan.** Default selection walks
|
||
registered backends in order; HIP registers before Vulkan, and the iGPU
|
||
dedup rule (PR #23897 workaround) keeps the first backend's device.
|
||
Verified live 2026-08-15: device list `[RPC0, ROCm0]`, Vulkan0 never
|
||
listed. Consequence: the Strix half runs on the ROCm backend — better
|
||
prefill, ~3% slower decode than Vulkan0 (measured on this box).
|
||
A second head build (`/home/dank/llama.cpp/build-vulkan-rpc`, HIP off,
|
||
Vulkan on, RPC on) was built the same night for an A/B; with it the
|
||
device list becomes `[RPC0, Vulkan0]`.
|
||
- **Default split is proportional to *reported free memory*, which is wrong
|
||
here.** The Strix reports up to 96 GiB (GTT ceiling) vs the 3090's 24 GB,
|
||
so the default lands ~20/80 in favor of the slow device (verified with a
|
||
4B smoke test: 4 of 36 layers on the 3090). We pin `--tensor-split 1,1`
|
||
(50/50) explicitly.
|
||
- **`-dio` (direct I/O) on the head.** Large model + RPC load hangs without
|
||
it (visorcraft Strix finding; matches our 28 GiB load-stall history).
|
||
- **Version skew hangs the handshake.** Worker built at the identical
|
||
commit (9b05354ec / b10433) in a worktree
|
||
`/home/dank/llama-cpp-rpc` on Dankserver — production b10121 build
|
||
untouched.
|
||
|
||
Smoke test (2026-08-15, 4B model, 0.2/0.8 split, production untouched):
|
||
worker VRAM rose ~819 MiB (real split), 95 t/s prefill, 58.8 t/s decode on
|
||
the tiny model. RTT on the link: ~0.3–0.5 ms.
|
||
|
||
Full design, scripts, and morning runbook:
|
||
[Qwen3.8 PP Deployment](Pipeline%20Parallelism/Qwen3.8%20PP%20Deployment.md).
|
||
|
||
### 2. Speculative-decoding tuning (same bits, free tokens)
|
||
|
||
- **MTP `n_max`.** Production leaves `--spec-draft-n-max` unset. Our matrix
|
||
(Vulkan, 16k) found 13.3 t/s at the tested setting vs 7.2 without spec;
|
||
the published Strix tuning curve peaks at n_max 3–4 (30.3 t/s at 4 vs
|
||
28.9 at 3 vs 29.9 at 5). The PP env sets **4** explicitly; the matrix
|
||
re-tests 3/4/5 on the real split.
|
||
- **Q4_0 draft head.** The published Strix card measured a standalone
|
||
Q4_0 MTP head ~6–10% faster than Q8_0 for the same acceptance (the draft
|
||
head is a 248k-vocab output projection; cheaper dequant, same accuracy).
|
||
The bundled head inside UD-Q8_K_XL is Q8-class; the standalone
|
||
`ggml-org/mtp-Qwen3.8-27B-Q4_0.gguf` (1.56 GB) is being downloaded for
|
||
an A/B via `--spec-draft-model`.
|
||
- **Ngram-mod fallback stays.** It is the only spec type that survived
|
||
parallel load (DSpark crashed ROCm, DFlash corrupts multi-slot).
|
||
|
||
### 3. Same-bit quant A/B: UD-Q8_K_XL vs plain Q8_0
|
||
|
||
Both are 8-bit. UD (unsloth "dynamic") mixes block sizes per tensor; plain
|
||
`Q8_0` uses uniform 32-element blocks. Dequant of a smaller block costs
|
||
more ALUs per element; on a memory-bound GPU the difference is usually
|
||
small, but Strix's iGPU has tight ALU headroom. `ggml-org/Qwen3.8-27B-GGUF`
|
||
ships a Q8_0 (29.12 GB). One restart to test; if it wins, same quality at
|
||
zero quality cost.
|
||
|
||
### 4. Prefill is the agentic bottleneck — route it right
|
||
|
||
Every agentic turn re-prefills the whole conversation (no prefix caching for
|
||
hybrid state, upstream). Levers:
|
||
|
||
- **Backend choice per workload.** Measured @64k on this box (b10433):
|
||
ROCm 410 t/s vs Vulkan 302 t/s prefill; Vulkan 25.1 vs ROCm 24.3 t/s
|
||
decode. For chat-heavy use: Vulkan. For agentic/batch-heavy use: ROCm.
|
||
The PP head (ROCm side) gets this for free.
|
||
- **ubatch/threads-batch.** Production runs ubatch 256. Larger ubatches
|
||
(512/1024) fill the iGPU better during prefill; `--threads-batch` for
|
||
the CPU-bound tail. Untested at prefill scale on this box — matrix item.
|
||
- **KV q4_0** for very long contexts: halves the 16-layer KV (8 GiB → 4 GiB
|
||
at 262k). Only worth it if long-ctx attention is the bottleneck; q8_0 is
|
||
the safe default.
|
||
|
||
### 5. OS level (from the Improvements doc, re-verified 2026-08-15)
|
||
|
||
- **TTM `pages_limit` is still 3,995,281 pages ≈ 15.24 GiB** — verified
|
||
today, unchanged. The 96 GiB GTT budget only materializes if the TTM cap
|
||
is raised (kernel parameter, needs reboot). Until then the effective
|
||
ceiling is lower and the 28 GiB load-stall risk remains. **Highest-value
|
||
single OS change; schedule with the next reboot.**
|
||
- `iommu=pt` — verified present on the kernel cmdline. Done.
|
||
- **Vulkan ICD pinning** — `radeon_icd.json` exists; no `VK_ICD_FILENAMES`
|
||
or loader-layer env is set. Pinning the radeon ICD and disabling loader
|
||
layers (`VK_LOADER_LAYERS_DISABLE=all`) is the recommended Strix
|
||
hardening; testable as service env vars, zero risk.
|
||
- **Power** — GPU DPM is already `performance`, forced level `auto`. The
|
||
`power_profile` sysfs file assumed in the Improvements doc does not exist
|
||
on this machine (checked 2026-08-15); that item is N/A here.
|
||
- Threads: 12 of 16 cores in production. Community reports 12–16 are
|
||
equivalent; leave at 12.
|
||
|
||
### 6. llama.cpp version watch
|
||
|
||
b10433 (2026-08-15) is adequate. Through b10451 there are no
|
||
spec-decoding fixes that help us; DFlash's multi-slot corruption is still
|
||
open upstream (fix #27117 pending). If that lands, DFlash/DSpark become
|
||
re-testable candidates (DSpark was the best single-machine draft before the
|
||
ROCm crash). No rebuild planned until then.
|
||
|
||
## Quantization landscape (reference)
|
||
|
||
The constraint is to keep UD-Q8_K_XL. This section records what exists so
|
||
future decisions are informed.
|
||
|
||
| Format | Size | Bits/weight | Notes |
|
||
|--------|------|-------------|-------|
|
||
| **UD-Q8_K_XL (unsloth)** | 31.5 GB | ~8.5 (mixed) | **Current.** Dynamic mixed-block; bundled MTP head; our baseline. |
|
||
| Q8_0 (ggml-org, bartowski) | 29.1 GB | 8.0 | Uniform 32-block. Same-bit A/B candidate (lever 3). |
|
||
| UD-Q6_K_XL (unsloth) | ~22 GB | ~6.5 | Below the 8-bit floor; documented, not recommended under the constraint. |
|
||
| Q6_K | ~21.5 GB | 6.6 | Same tier as above. |
|
||
| **ROCmFP4 "STRIX" (ROCmFPX fork)** | 13.3–13.8 GB | 4.25–4.39 | Strix-specific NVFP4-style format, ROCm backend only, requires a forked llama.cpp. Published: 30.3 t/s vs 10.7 for Q4_K_M (2.83×) at 8k with MTP n_max≈4; PPL ≈ Q4_K_M. **This is the "fast path" — and it is a quant downgrade plus a fork dependency that breaks the Vulkan production path.** Kept here as the escape hatch if the 8-bit speed target cannot be met. |
|
||
| NVFP4 (Blackwell) | ~14 GB | 4.5 | Hardware path for RTX 50-series. Irrelevant to 3090/Strix. |
|
||
|
||
If the 8-bit PP result (lever 1) is disappointing, the decision order is:
|
||
push PP split/latency tuning → ROCmFP4 as a *second* service (A/B, not a
|
||
replacement) → revisit the constraint with Frank.
|
||
|
||
## Morning test matrix (2026-08-16)
|
||
|
||
Policy: stop both production services, run tests, restore. Each config =
|
||
one head restart (~10 min load incl. weight stream; ~2 min with the worker
|
||
tensor cache warm).
|
||
|
||
| # | Config | Measures |
|
||
|---|--------|----------|
|
||
| A | PP split 1,1, ROCm head, MTP n_max=4 + ngram | baseline PP |
|
||
| B | A + `SPEC_NONE=1` | raw (no spec) PP speed |
|
||
| C | A but head build `build-vulkan-rpc` (Vulkan side) | backend A/B |
|
||
| D | A + `--spec-draft-model` Q4_0 MTP file | draft-head A/B |
|
||
| E | A with n_max 3 and 5 | n_max curve on the split |
|
||
| F | A, plain Q8_0 model file | same-bit A/B |
|
||
| G | PP default proportional split (no `--tensor-split`) | confirms the 20/80 handicap |
|
||
| Ref | production frank-default-large (Vulkan, 2 slots) | the bar to beat |
|
||
|
||
Per config: `pp-test.sh bench 4` (4k prompt / 256 out) and `bench 16`
|
||
(16k / 256). Report: decode t/s (server print_timing), prefill t/s, MTP
|
||
acceptance, worker VRAM, head GTT usage.
|
||
|
||
Success bar: **≥ 30 t/s at 4k and ≥ 20 t/s at 16k decode** (vs 25 t/s
|
||
production @16k) with the same 8-bit model.
|
||
|
||
## Sources
|
||
|
||
- GGUF header dump of the production model (this machine, 2026-08-15).
|
||
- Qwen3.8-27B model card + `ggml-org/Qwen3.8-27B-GGUF` /
|
||
`unsloth/Qwen3.8-27B-GGUF` repo listings (Hugging Face).
|
||
- kingjones `Qwen3.8-27B-ROCmFP4-GGUF` Strix tuning card (MTP n_max curve,
|
||
Q4_0 vs Q8_0 draft head, ROCmFP4 numbers).
|
||
- visorcraft `strix-halo-llm-perf` (Vulkan tg / ROCm prefill, ICD pinning,
|
||
`-dio` for RPC).
|
||
- llama.cpp b10433 source: `src/llama.cpp` (device selection),
|
||
`common/speculative.cpp` (MTP/ngram-mod), `ggml/src/ggml-rpc` (protocol).
|
||
- Local benchmarks: [frank-default-large-benchmark](frank-default-large-benchmark.md),
|
||
[qwen3.8-spec-matrix](qwen3.8-spec-matrix.md).
|