Files

T

Frank a5fe045b58 docs(vault): one fact, one owner — repair the doubled pages, single-source the facts

Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.

2026-10-19 05:00:43 -07:00

13 KiB

Raw Blame History

type, description, tags, timestamp

type

description

Qwen3.8-27B on Strix Halo: Speed Without Lowering the Quant

This page answers one question: how do we make Qwen3.8-27B serve faster on Dankclaw (Strix Halo, Radeon 8060S) while keeping UD-Q8_K_XL? It records the model's real architecture, the levers that are available, their evidence, and the test matrix planned for 2026-08-16 morning.

Speculative-decoding matrix
frank-dragoon — the endpoint this host became; its Eras and Laws carry the backend and spec-decode measurements.
Dankclaw Improvements (OS-level items)
Pipeline Parallelism: Qwen3.8 PP Deployment

TL;DR — ranked levers

#	Lever	Expected gain	Cost	Status
1	Pipeline-parallel to the RTX 3090 (RPC)	~1.5–2× decode	2.5 GbE link, both services stop for test	Kit ready, test 08-16
2	MTP `n_max` explicit 4–5 + Q4_0 draft head	+5–15% on the spec share	restart, 1.56 GB file	test 08-16
3	Plain `Q8_0` A/B vs UD-Q8_K_XL (same 8 bits)	dequant ALUs may differ	restart, 29 GB file	test 08-16
4	Prefill path: ROCm backend + tuned ubatch/threads-batch	2–3× prefill for agentic turns	per-workload backend choice	partial data
5	TTM `pages_limit` raised	removes the hard VRAM ceiling; the full 262k f16 KV plus the Q8 AGENT payload, MTP draft, and mmproj fit in one slot	modprobe, set live 2026-08-25	applied — value and file: Dankclaw System Specs (Memory)
6	Vulkan ICD pinning + loader-layer disable	robustness; small	service env vars	test 08-16

Everything else (quantization downgrade, NVFP4, larger drafts) is documented below for completeness but is out of scope per the "no lowering" constraint.

What the model actually is

Verified from the GGUF header (/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL.gguf, dumped 2026-08-15) and the official model card.

Architecture: qwen35 / Qwen3.8-27B, hybrid attention.
64 main layers + 1 MTP (nextn) layer = 65 GGUF blocks.
48 Gated DeltaNet (linear/recurrent) layers + 16 gated full-attention layers, in a 3:1 pattern (full_attention_interval = 4).
Hidden 5120, FFN 17408, 24 Q heads / 4 KV heads, head dim 256.
Native context 262144 (extendable per the card; production runs 262k/slot).
GDN layers keep a fixed-size recurrent state (constant per layer, not per token). Only the 16 full-attention layers grow with context.
KV per token (full-attention part only): 16 layers × 2 (K,V) × 4 heads × 256 dim × 1 byte (q8_0) = 32 KiB/token. At 65536 ctx ≈ 2 GiB; at 262144 ctx ≈ 8 GiB.
Sampling defaults (card, thinking mode on by default): temperature 1.0, top-k 20, top-p 0.95, min-p 0.0. Production already matches.

Consequences of the hybrid design:

Decode is cheap in memory. Attention cost scales with only 16 layers; the 48 GDN layers are O(1) per token. Long-context decode degrades far slower than a dense transformer.
Prompt cache reuse is unavailable upstream. Recurrent/SSM state cannot be prefix-cached the way attention KV can. Every agentic turn re-prefills the whole conversation. Prefill is the hidden cost of agentic use, and it is the dominant cost we have not yet optimized.
MTP is native and strong. The model ships a nextn head; on Vulkan at 16k it roughly doubled the no-spec rate (the pair of figures is qwen3.8-spec-matrix's realistic-prompt table).

Baseline this page optimized (2026-08-15)

frank-dragoon then ran llama.cpp on the iGPU with the UD-Q8_K_XL target (bundled MTP head), --spec-default ngram-mod under a draft-mtp first link, two 262144 slots, q8_0 KV, batch 8192 / ubatch 256, threads 12. That era's ledger is frank-dragoon's; what this page keeps is the shape of the problem: the service occupied 28 GiB of the then 96 GiB firmware VRAM carve, ~74 GiB idle, decode held ~25 t/s at 16k and collapsed at 262k depth (qwen3.8-spec-matrix §High-Context Backend Comparison), and prefill was the unoptimized agentic cost (section 4).

The levers, in detail

1. Pipeline-parallel to the RTX 3090 (biggest win)

The 3090 on Dankserver had headroom that night (it carried the day's small-model service with a couple of GiB free — that card's eras are frank-zealot's). llama.cpp RPC splits the model by layers: each device runs its layer range, hidden states cross the 2.5 GbE ethernet once per token. The per-position wire bill is measured in Pipeline Parallelism/Ethernet Bandwidth Math §0; at ~0.5 ms LAN RTT the bandwidth cost is negligible — the split is latency-bound, not bandwidth-bound.

Why this is a real speedup and not just "two GPUs": the 3090 does matmuls roughly 2–3× faster per layer than the 8060S (936 vs 512 GB/s memory, 80 vs 40 CUs). A 50/50 layer split balances per-layer latency, so each token step takes ≈ max(half-on-3090, half-on-Strix) + one RTT instead of all-64-layers-on-Strix.

Every decision this led to is written once, in Pipeline Parallelism/Qwen3.8 PP Deployment §Gotchas (with the 4B smoke test that proved each): head on the box that already holds the model file, no --device on the head (setting it silently drops RPC), the fixed HIP-before-Vulkan device selection, the reported-free-memory split trap, the identical-build requirement, and -dio on the head — the last one visorcraft's Strix finding, matching our own 28 GiB load-stall history. Scripts and the morning runbook are on the same page.

2. Speculative-decoding tuning (same bits, free tokens)

MTP n_max. Production left --spec-draft-n-max unset; the fleet matrix (Vulkan, 16k, realistic prompt) found the depth-5 peak against the no-spec baseline — both figures are qwen3.8-spec-matrix's realistic-prompt table. The published Strix tuning curve peaks at n_max 3–4 (30.3 t/s at 4 vs 28.9 at 3 vs 29.9 at 5). The PP env set 4 explicitly; the matrix re-tested 3/4/5 on the real split.
Q4_0 draft head. The published Strix card measured a standalone Q4_0 MTP head ~6–10% faster than Q8_0 for the same acceptance (the draft head is a 248k-vocab output projection; cheaper dequant, same accuracy). The bundled head inside UD-Q8_K_XL is Q8-class; the standalone ggml-org/mtp-Qwen3.8-27B-Q4_0.gguf (1.56 GB) is being downloaded for an A/B via --spec-draft-model.
Ngram-mod fallback stays. It is the only spec type that survived parallel load; the DSpark ROCm fault and the DFlash multi-slot corruption are timed and explained on qwen3.8-spec-matrix.

3. Same-bit quant A/B: UD-Q8_K_XL vs plain Q8_0

Both are 8-bit. UD (unsloth "dynamic") mixes block sizes per tensor; plain Q8_0 uses uniform 32-element blocks. Dequant of a smaller block costs more ALUs per element; on a memory-bound GPU the difference is usually small, but Strix's iGPU has tight ALU headroom. ggml-org/Qwen3.8-27B-GGUF ships a Q8_0 (29.12 GB). One restart to test; if it wins, same quality at zero quality cost.

4. Prefill is the agentic bottleneck — route it right

Every agentic turn re-prefills the whole conversation (no prefix caching for hybrid state, upstream). Levers:

Backend choice per workload. Measured @64k on this box (b10433, the era frank-dragoon owns): ROCm 410 t/s vs Vulkan 302 t/s prefill; Vulkan 25.1 vs ROCm 24.3 t/s decode. For chat-heavy use: Vulkan. For agentic/batch-heavy use: ROCm. The PP head (ROCm side) gets this for free.
ubatch/threads-batch. Production runs ubatch 256. Larger ubatches (512/1024) fill the iGPU better during prefill; --threads-batch for the CPU-bound tail. Untested at prefill scale on this box — matrix item.
KV q4_0 for very long contexts: halves the 16-layer KV (8 GiB → 4 GiB at 262k). Only worth it if long-ctx attention is the bottleneck; q8_0 is the safe default.

5. OS level (from the Improvements doc, re-verified 2026-08-15)

TTM pages_limit — the raise itself (value, live write, modprobe file, date) is one row on Dankclaw System Specs (Memory). What matters to this page is the mechanism: the driver-reported 96 GiB VRAM carve-out (VGM/SVM pool) was never the ceiling — the TTM kernel page map was, and until it was raised a 262k f16 KV cache plus a ~28 GB payload could not load in one slot.
Boot line — IOMMU policy on this host is dankclaw-iommu-translation-off's and Dankclaw System Specs's; it has changed twice since this lever was written, so no era-scoped reading of /proc/cmdline belongs here.
Vulkan ICD pinning — radeon_icd.json exists; no VK_ICD_FILENAMES or loader-layer env is set. Pinning the radeon ICD and disabling loader layers (VK_LOADER_LAYERS_DISABLE=all) is the recommended Strix hardening; testable as service env vars, zero risk.
Power — GPU DPM is already performance, forced level auto. The power_profile sysfs file assumed in the Improvements doc does not exist on this machine (checked 2026-08-15); that item is N/A here.
Threads: 12 of 16 cores in production. Community reports 12–16 are equivalent; leave at 12.

6. llama.cpp version watch

b10433 (2026-08-15) is adequate. Through b10451 there are no spec-decoding fixes that help us; DFlash's multi-slot corruption is still open upstream (fix #27117 pending). If that lands, DFlash/DSpark become re-testable candidates (DSpark was the best single-machine draft before the ROCm crash). No rebuild planned until then.

Quantization landscape (reference)

The constraint is to keep UD-Q8_K_XL. This section records what exists so future decisions are informed.

Format	Size	Bits/weight	Notes
UD-Q8_K_XL (unsloth)	31.5 GB	~8.5 (mixed)	Current. Dynamic mixed-block; bundled MTP head; our baseline.
Q8_0 (ggml-org, bartowski)	29.1 GB	8.0	Uniform 32-block. Same-bit A/B candidate (lever 3).
UD-Q6_K_XL (unsloth)	~22 GB	~6.5	Below the 8-bit floor; documented, not recommended under the constraint.
Q6_K	~21.5 GB	6.6	Same tier as above.
ROCmFP4 "STRIX" (ROCmFPX fork)	13.3–13.8 GB	4.25–4.39	Strix-specific NVFP4-style format, ROCm backend only, requires a forked llama.cpp. Published: 30.3 t/s vs 10.7 for Q4_K_M (2.83×) at 8k with MTP n_max≈4; PPL ≈ Q4_K_M. This is the "fast path" — and it is a quant downgrade plus a fork dependency that breaks the Vulkan production path. Kept here as the escape hatch if the 8-bit speed target cannot be met.
NVFP4 (Blackwell)	~14 GB	4.5	Hardware path for RTX 50-series. Irrelevant to 3090/Strix.

If the 8-bit PP result (lever 1) is disappointing, the decision order is: push PP split/latency tuning → ROCmFP4 as a second service (A/B, not a replacement) → revisit the constraint with Frank.

Morning test matrix (2026-08-16)

Policy: stop both production services, run tests, restore. Each config = one head restart (~10 min load incl. weight stream; ~2 min with the worker tensor cache warm).

#	Config	Measures
A	PP split 1,1, ROCm head, MTP n_max=4 + ngram	baseline PP
B	A + `SPEC_NONE=1`	raw (no spec) PP speed
C	A but head build `build-vulkan-rpc` (Vulkan side)	backend A/B
D	A + `--spec-draft-model` Q4_0 MTP file	draft-head A/B
E	A with n_max 3 and 5	n_max curve on the split
F	A, plain Q8_0 model file	same-bit A/B
G	PP default proportional split (no `--tensor-split`)	confirms the 20/80 handicap
Ref	production frank-dragoon (Vulkan, 2 slots)	the bar to beat

Per config: pp-test.sh bench 4 (4k prompt / 256 out) and bench 16 (16k / 256). Report: decode t/s (server print_timing), prefill t/s, MTP acceptance, worker VRAM, head GTT usage.

Success bar: ≥ 30 t/s at 4k and ≥ 20 t/s at 16k decode, against the production @16k bar in the Baseline section, with the same 8-bit model.

Sources

GGUF header dump of the production model (this machine, 2026-08-15).
Qwen3.8-27B model card + ggml-org/Qwen3.8-27B-GGUF / unsloth/Qwen3.8-27B-GGUF repo listings (Hugging Face).
kingjones Qwen3.8-27B-ROCmFP4-GGUF Strix tuning card (MTP n_max curve, Q4_0 vs Q8_0 draft head, ROCmFP4 numbers).
visorcraft strix-halo-llm-perf (Vulkan tg / ROCm prefill, ICD pinning, -dio for RPC).
llama.cpp b10433 source: src/llama.cpp (device selection), common/speculative.cpp (MTP/ngram-mod), ggml/src/ggml-rpc (protocol).
Local benchmarks: the measurement record lives on frank-dragoon (Eras and Laws) and in qwen3.8-spec-matrix.

13 KiB Raw Blame History Unescape Escape