PP kit + Strix research: Qwen3.8-27B RPC split (Dankclaw head / Dankserver worker)

- scripts: serve-pp-head/worker, envs, systemd units, pp-test.sh orchestrator, pp-bench.py - head design verified in b10433 source + live 4B smoke test: * no --device with --rpc (RPC device silently dropped otherwise) * default device selection = [RPC0, ROCm0] (HIP registers before Vulkan) * default split proportional to reported free mem = ~20/80 here, pin 1,1 * -dio required for large model over RPC - docs: qwen3.8-strix-optimizations (ranked no-quant-downgrade levers, quant landscape, 2026-08-16 test matrix), PP deployment runbook, index updates, Dankclaw Improvements corrections (Vulkan0, no power_profile sysfs) - worker built at 9b05354ec in /home/dank/llama-cpp-rpc worktree (CUDA+RPC) - second head build build-vulkan-rpc (HIP off) for backend A/B
2026-08-16 05:40:27 -07:00
1 parent c775077016
commit 9544c36afc
@@ -6,7 +6,23 @@ inference runtime tuning, and server/OS/hardware settings. Each item lists a
 priority and the exact action.

 Dates: researched 2026-08. Hardware is verified on the live machine unless
-marked "verify first".
+marked "verify first". Re-verified 2026-08-15/16; corrections marked below.
+
+## Corrections (re-verified 2026-08-15/16)
+
+- **Backend**: production is **Vulkan0**, not ROCm. Measured on this box
+  (b10433): Vulkan 25.13 t/s vs ROCm 24.33 t/s decode at 16k; at 262k ctx
+  Vulkan decodes 48% faster (5.68 vs 3.83 t/s). ROCm wins prefill (410 vs
+  302 t/s @64k). The "HIP out-performs Vulkan for decode" claim did not
+  hold on the 8060S. See [[frank-default-large-benchmark]].
+- **Power profile**: the `/sys/class/drm/card0/device/power_profile` file
+  does not exist on this machine (checked 2026-08-15). GPU DPM is already
+  `performance`, forced level `auto`. This item is N/A.
+- **KV cache**: production runs `q8_0` K **and** `q8_0` V (both quantized).
+- **Spec**: DFlash/DSpark are retired. Production is `--spec-default`
+  (ngram-mod) with draft-mtp first. See [[qwen3.8-spec-matrix]].
+- **TTM**: still 15.24 GiB as of 2026-08-16 — the modprobe fix is still
+  pending a reboot.

 ## Verified Memory Picture (measured 2026-08-11)

@@ -47,8 +63,8 @@ Verified against the running service and llama.cpp source.

 | Change | Recommendation | Priority |
 | ------ | -------------- | -------- |
-| Backend | Keep HIP/ROCm. HIP out-performs Vulkan for decode on RDNA3.5/4 iGPU. | Done |
-| KV cache | Keep `q8_0` K / `f16` V. q8_0 halves memory vs f16 with negligible loss. Try `q4_0`/`iq4_nl` only if more context is needed. | Keep |
+| Backend | **Vulkan0** (2026-08-15, corrected — see Corrections). ROCm only for prefill-heavy workloads. | Done |
+| KV cache | `q8_0` K / `q8_0` V in production. Try `q4_0` only if more context is needed. | Keep |
 | Flash attention | Leave `--flash-attn auto`. It helps prompt processing, not decode. | Keep |
 | Load mode | Start with mmap (default). Enable `--mlock` only if page-fault stutter appears. | Optional |
 | Threads | Use `--threads 12 --threads-batch 16 --threads-http 1` for 16 Zen5 cores + 3 slots. Frees cores for the ROCm driver. | Medium |
@@ -62,7 +78,7 @@ All require root and a reboot for the modprobe change.
 | Setting | Action | Effect | Priority |
 | ------- | ------ | ------ | -------- |
 | TTM GPU pool | Set `amd-ttm --set 96` (write `/etc/modprobe.d/ttm.conf`), reboot | Lifts the 15 GiB TTM cap toward full RAM; more headroom for GPU buffers | High |
-| Power profile | `echo performance > /sys/class/drm/card0/device/power_profile` | Sustained inference speed; apply at boot via a udev/systemd unit | Medium |
+| Power profile | **N/A on this machine** (2026-08-15): no `power_profile` sysfs; GPU DPM already `performance` | none | ~~Medium~~ |
 | NVMe scheduler | Already `none` (`[none]` confirmed) | No change | Done |
 | BIOS VGM | Set Variable Graphics Memory / UMA to "High" in the EVO-X2 BIOS | Max dedicated VRAM for the GPU carve-out; verify in the live BIOS | Verify |
 | Firmware/EC | Check GMKtec for a newer EC/fan-curve firmware | Better sustained thermal behavior | Verify |
@@ -0,0 +1,137 @@
+# Qwen3.8-27B Pipeline-Parallel Deployment (Dankclaw + Dankserver)
+
+Split Qwen3.8-27B UD-Q8_K_XL across the Strix Halo iGPU (head) and the RTX
+3090 (worker) using llama.cpp RPC. This page is the design record and the
+morning-of runbook. Research background:
+[Qwen3.8 Strix optimizations](../../qwen3.8-strix-optimizations.md).
+
+## Topology
+
+```
+Dankclaw (head, 192.168.0.143)          Dankserver (worker, 192.168.0.235)
+llama-server b10433                      ggml-rpc-server b10433
+  --rpc 192.168.0.235:50052                 --host 192.168.0.235 --port 50052
+  model file (31.5 GB)                     -d CUDA0 -c (tensor cache)
+  [RPC0, ROCm0|Vulkan0]  layers            RTX 3090: layers 0..~31
+  Strix:    layers ~32..64 + lm_head
+  hidden state ~20 KiB crosses the link once per token, each direction
+```
+
+- **Layer-sequential split** (true pipeline parallelism): layers 0..N run on
+  the worker, the rest on the head, in order. One hidden-state handoff per
+  token step. RTT-bound, not bandwidth-bound.
+- **Why the head is on Dankclaw:** the model file is already there; the
+  worker streams its tensor share at load (no 31.5 GB copy to the server).
+- **Both sides must run the identical build** (b10433 / 9b05354ec). The
+  worker build lives in a worktree `/home/dank/llama-cpp-rpc` on Dankserver
+  so the production b10121 build stays untouched. A version skew hangs the
+  RPC handshake with no error.
+
+## Files
+
+| File | Machine | Purpose |
+|------|---------|---------|
+| `scripts/frank-pp-worker.env` | server | worker tunables (bind, port, device, cache) |
+| `scripts/serve-pp-worker.sh` | server | runs `ggml-rpc-server` from the b10433 worktree build |
+| `scripts/frank-pp-head.env` | claw | head tunables (split, ctx, spec, KV, I/O) |
+| `scripts/serve-pp-head.sh` | claw | runs `llama-server --rpc ...` (see "Gotchas") |
+| `scripts/frank-pp-head.service` / `frank-pp-worker.service` | both | optional systemd units (manual install) |
+| `scripts/pp-test.sh` | any box with ssh to both | test orchestrator (preflight/start/verify/bench/stop-pp/restore) |
+| `scripts/pp-bench.py` | claw | API driver; reports server-side `print_timing` |
+
+## Gotchas (all verified in b10433 source / live tests, 2026-08-15)
+
+1. **No `--device` on the head.** Setting `params.devices` makes
+   `llama_prepare_model_devices` use exactly that list; the RPC device is
+   silently dropped and `--rpc` becomes a no-op. Omit it; default selection
+   puts RPC first.
+2. **The iGPU side is ROCm by default.** Backends register in fixed order
+   (HIP before Vulkan); the iGPU dedup rule keeps the first backend's view.
+   Live check showed `[RPC0, ROCm0]`. For a Vulkan side, use the second
+   head build `/home/dank/llama.cpp/build-vulkan-rpc` (HIP off) — set
+   `HEAD_BIN` in the env. Trade: ROCm side = faster prefill, ~3% slower
+   decode (measured on this box).
+3. **Pin `--tensor-split 1,1`.** The default split is proportional to
+   *reported free memory*: the Strix reports up to 96 GiB (GTT) vs the
+   3090's 24 GB, so the default lands ~20/80 on the slow device (verified
+   with a 4B smoke test). 1,1 = 50/50 = balanced per-layer latency.
+4. **`-dio` on the head.** Large model + RPC hangs at load without direct
+   I/O.
+5. **RPC is unauthenticated and unencrypted.** LAN only. The worker binds
+   0.0.0.0-equivalent by design (`--host 192.168.0.235`).
+6. **Env var names.** Never `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`
+   (HIP runtime reads them; the 2026-08-13 `TEMP` incident).
+7. **Page-cache state confounds loads.** `pp-test.sh start` drops caches on
+   the head before load (warm-cache loads crash fast; cold loads survive).
+
+## Runbook (morning of 2026-08-16)
+
+Run from any box with ssh access to both (repo path on claw:
+`/home/dank/vllmScripts`, on server: `/home/dank/vLLM`).
+
+```
+# 0. one-time: pull the kit on both boxes
+ssh dank@dankclaw   'cd /home/dank/vllmScripts  && git pull'
+ssh dank@dankserver 'cd /home/dank/vLLM && git pull'
+
+# 1. checks (read-only)
+./scripts/pp-test.sh preflight
+
+# 2. full run: stop prod -> start PP -> verify split -> bench -> restore
+FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh all
+
+# or step by step:
+FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh start
+./scripts/pp-test.sh verify          # layer split + worker VRAM
+./scripts/pp-test.sh bench 4 256 2   # ~4k prompt, 256 out, 2 reps
+./scripts/pp-test.sh bench 16 256 1
+FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh stop-pp
+FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh restore
+```
+
+Config variants: edit `scripts/frank-pp-head.env` on the claw (git commit +
+pull if the bench runs from the repo), then re-run `start` + `bench`:
+
+- No spec baseline: `SPEC_NONE=1`
+- Vulkan head side: `HEAD_BIN=/home/dank/llama.cpp/build-vulkan-rpc/bin/llama-server`
+- Q4_0 MTP draft head: add `--spec-draft-model /home/dank/models/Qwen3.8-27B-MTP-Q4_0.gguf` to the serve script's SPEC block (env hook: see `SPEC_DRAFT_MODEL` below)
+- n_max sweep: `SPEC_DRAFT_N_MAX=3|4|5`
+- Plain Q8_0: `MODEL_FILE=Qwen3.8-27B-Q8_0.gguf` (file must be on the claw)
+
+Verification checklist after each `start`:
+
+1. `verify` shows `using device RPC0 ... using device ROCm0|Vulkan0`.
+2. Layer assignment lines (`-lv 10`): worker gets the first ~32 layers at a
+   1,1 split.
+3. `nvidia-smi` on the server shows ~15–16 GB used by the worker
+   (production small service + worker share).
+4. Head GTT usage (rocm-smi) well under 96 GiB; model fully on devices
+   (`--gpu-layers all` offloads everything; CPU holds embeddings/lm_head glue).
+
+## Expected numbers
+
+Per-layer: 3090 ≈ 2–3× faster than the 8060S. 50/50 split → each token
+step ≈ max(32 layers on 3090, 32 layers on Strix) + ~0.5 ms RTT.
+
+- Production reference: 25 t/s @16k (Vulkan, spec, single node).
+- Target: **≥ 30 t/s @4k, ≥ 20 t/s @16k** with spec.
+- Decode floor: with ngram/mod acceptance, spec multiplies the raw rate;
+  raw (no spec) rate should clear ~15–18 t/s @16k for the target to hold.
+
+## Rollback
+
+`pp-test.sh restore` starts both production services. If a PP test wedges a
+box: `systemctl restart <unit>` after killing the stray processes
+(`pkill -x llama-server` / `pkill -x ggml-rpc-server` — never `pkill -f`
+with a pattern matching your own ssh command line).
+
+## Open questions for the test day
+
+1. Does the MTP head (which sits in the last layer group, iGPU side) draft
+   cleanly with the verify pass crossing the RPC boundary? Watch acceptance
+   in the head log (`draft acceptance = X`).
+2. Two parallel slots on the split (PARALLEL=2, CTX 65536): RPC is
+   serialized per compute — expect the 2-slot speedup to be weaker than
+   single-node. Measure, do not assume.
+3. Prefill parity: ROCm-side head should prefill ≥ single-node ROCm; if not,
+   the ubatch crossing the RPC boundary is the suspect.
@@ -6,7 +6,7 @@ tags:
  - pipeline-parallelism
  - distributed
  - index
-timestamp: 2026-08-14
+timestamp: 2026-08-15
 ---

 # Pipeline Parallelism
@@ -58,6 +58,14 @@ graph LR
 - [[Expected Performance]] — estimated tokens per second for a 27B Q8 split
  across an RTX 3090 and a Strix Halo NUC over 1/2.5/10 GbE.

+## Deployment
+
+- [[Qwen3.8 PP Deployment]] — the real Qwen3.8-27B split: head on Dankclaw
+  (Strix), worker on Dankserver (RTX 3090), b10433 on both sides, scripts in
+  `scripts/`, morning-of test runbook. Verified gotchas: no `--device` with
+  `--rpc`, default split is proportional to reported free memory (pin 1,1),
+  the iGPU side defaults to the ROCm backend.
+
 ## Sample Scripts

 - [[Sample Script Set]] — a two-node llama.cpp RPC split of
@@ -53,6 +53,7 @@ Vulkan0 backend, port 7770, PARALLEL=2]
 - [[Running Qwen3.6 27B FP8 on Dankclaw]] — Dense model (27B) on Strix Halo APU
 - [[frank-default-large-benchmark]] — backend/spec benchmark; chose MTP sidecar (2026-08-14) over DFlash (parallel premature-EOS bug)
 - [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL spec matrix; production = --spec-default (ngram-mod), PARALLEL=2, Vulkan0 backend (2026-08-15)
+- [[qwen3.8-strix-optimizations]] — Qwen3.8-27B Strix speed research: ranked levers that keep UD-Q8_K_XL (PP to 3090, MTP tuning, same-bit Q8_0 A/B, prefill path, OS), quant landscape, 2026-08-16 test matrix
 - [[frank-default-small-benchmark]] — ncmoe and batch/ubatch sweep that drove the Dankserver state
 - [[dankserver-vram-growth]] — VRAM/RAM growth investigation behind the recurring cuBLAS crash on frank-default-small
 - [[Dankclaw Improvements]] — researched enhancements: self-hosted agentic tools, runtime tuning, server/OS settings
@@ -71,3 +72,4 @@ Vulkan0 backend, port 7770, PARALLEL=2]
  - [[Pipeline Parallelism/Mixed-Vendor Experiments]] — NVIDIA + Strix Halo real-world runs
  - [[Pipeline Parallelism/Expected Performance]] — estimated tok/s over 1/2.5/10 GbE
  - [[Pipeline Parallelism/Sample Script Set]] — educational two-node Qwen3.6-27B Q8 split
+  - [[Pipeline Parallelism/Qwen3.8 PP Deployment]] — production design + runbook: Qwen3.8-27B head (Strix) / worker (RTX 3090) over RPC, b10433, scripts in scripts/
@@ -0,0 +1,247 @@
+# Qwen3.8-27B on Strix Halo: Speed Without Lowering the Quant
+
+This page answers one question: **how do we make Qwen3.8-27B serve faster on
+Dankclaw (Strix Halo, Radeon 8060S) while keeping UD-Q8_K_XL?** It records
+the model's real architecture, the levers that are available, their evidence,
+and the test matrix planned for 2026-08-16 morning.
+
+Related pages:
+- [Speculative-decoding matrix](qwen3.8-spec-matrix.md)
+- [Backend and spec benchmark](frank-default-large-benchmark.md)
+- [Dankclaw Improvements](Dankclaw%20Improvements.md) (OS-level items)
+- [Pipeline Parallelism: Qwen3.8 PP Deployment](Pipeline%20Parallelism/Qwen3.8%20PP%20Deployment.md)
+
+## TL;DR — ranked levers
+
+| # | Lever | Expected gain | Cost | Status |
+|---|-------|--------------|------|--------|
+| 1 | Pipeline-parallel to the RTX 3090 (RPC) | ~1.5–2× decode | 2.5 GbE link, both services stop for test | Kit ready, test 08-16 |
+| 2 | MTP `n_max` explicit 4–5 + Q4_0 draft head | +5–15% on the spec share | restart, 1.56 GB file | test 08-16 |
+| 3 | Plain `Q8_0` A/B vs UD-Q8_K_XL (same 8 bits) | dequant ALUs may differ | restart, 29 GB file | test 08-16 |
+| 4 | Prefill path: ROCm backend + tuned ubatch/threads-batch | 2–3× prefill for agentic turns | per-workload backend choice | partial data |
+| 5 | TTM `pages_limit` raise (15.2 GiB → 90+) | removes a hard VRAM ceiling, stable 96 GiB budget | reboot | pending |
+| 6 | Vulkan ICD pinning + loader-layer disable | robustness; small | service env vars | test 08-16 |
+
+Everything else (quantization downgrade, NVFP4, larger drafts) is documented
+below for completeness but is out of scope per the "no lowering" constraint.
+
+## What the model actually is
+
+Verified from the GGUF header (`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL.gguf`,
+dumped 2026-08-15) and the official model card.
+
+- Architecture: `qwen35` / Qwen3.8-27B, **hybrid attention**.
+- 64 main layers + 1 MTP (nextn) layer = 65 GGUF blocks.
+- 48 Gated DeltaNet (linear/recurrent) layers + 16 gated full-attention
+  layers, in a 3:1 pattern (`full_attention_interval = 4`).
+- Hidden 5120, FFN 17408, 24 Q heads / 4 KV heads, head dim 256.
+- Native context 262144 (extendable per the card; production runs 262k/slot).
+- GDN layers keep a **fixed-size** recurrent state (constant per layer, not
+  per token). Only the 16 full-attention layers grow with context.
+- KV per token (full-attention part only): 16 layers × 2 (K,V) × 4 heads ×
+  256 dim × 1 byte (q8_0) = **32 KiB/token**. At 65536 ctx ≈ 2 GiB; at
+  262144 ctx ≈ 8 GiB.
+- Sampling defaults (card, thinking mode on by default): temperature 1.0,
+  top-k 20, top-p 0.95, min-p 0.0. Production already matches.
+
+Consequences of the hybrid design:
+
+1. **Decode is cheap in memory.** Attention cost scales with only 16 layers;
+   the 48 GDN layers are O(1) per token. Long-context decode degrades far
+   slower than a dense transformer.
+2. **Prompt cache reuse is unavailable upstream.** Recurrent/SSM state
+   cannot be prefix-cached the way attention KV can. Every agentic turn
+   re-prefills the whole conversation. **Prefill is the hidden cost of
+   agentic use**, and it is the dominant cost we have not yet optimized.
+3. **MTP is native and strong.** The model ships a nextn head; our matrix
+   measured 13.3 t/s vs 7.2 t/s (no spec) at 16k on Vulkan.
+
+## Current production state (baseline)
+
+`frank-default-large` (Dankclaw, Vulkan0, 2026-08-15):
+
+- Model: UD-Q8_K_XL, 31.5 GB, bundled MTP head.
+- `--spec-default` (ngram-mod) with draft-mtp first in the chain.
+- 2 parallel slots × 262144 ctx, KV q8_0, batch 8192, ubatch 256,
+  threads 12, `-dio` off, `--fit off`.
+- Measured: ~25 t/s at 16k ctx; 5.68 t/s at 262k ctx; prefill 302 t/s @64k
+  (Vulkan) vs 410 t/s @64k (ROCm, same build).
+- GPU: 28 GiB of 96 GiB GTT used. **74 GiB sits idle.**
+
+## The levers, in detail
+
+### 1. Pipeline-parallel to the RTX 3090 (biggest win)
+
+The 3090 on Dankserver is idle (serves a 22 GB MoE small-model service with
+~2.4 GB free). llama.cpp RPC splits the model by **layers**: each device
+runs its layer range, hidden states cross the 2.5 GbE ethernet once per
+token. The per-token transfer is ~41 KiB (5120-wide f32 hidden state, both
+directions); at ~0.5 ms LAN RTT the bandwidth cost is negligible — the split
+is latency-bound, not bandwidth-bound (see
+[Ethernet Bandwidth Math](Pipeline%20Parallelism/Ethernet%20Bandwidth%20Math.md)).
+
+Why this is a real speedup and not just "two GPUs": the 3090 does
+matmuls roughly 2–3× faster per layer than the 8060S (936 vs 512 GB/s
+memory, 80 vs 40 CUs). A 50/50 layer split balances per-layer latency, so
+each token step takes ≈ max(half-on-3090, half-on-Strix) + one RTT instead
+of all-64-layers-on-Strix.
+
+Design decisions (verified in b10433 source, 2026-08-15):
+
+- **Head on Dankclaw, worker on Dankserver.** The 31.5 GB model file stays
+  on the head; the worker receives its tensor share over the wire at load
+  (with `-c` it caches the tensors locally after the first load).
+- **Head must NOT pass `--device`.** When `params.devices` is set,
+  `llama_prepare_model_devices` builds the device list from exactly those
+  devices and the RPC device is silently dropped (`--rpc` becomes a no-op).
+  Default device selection inserts RPC servers **at the front** of the list.
+- **The local iGPU side runs on ROCm, not Vulkan.** Default selection walks
+  registered backends in order; HIP registers before Vulkan, and the iGPU
+  dedup rule (PR #23897 workaround) keeps the first backend's device.
+  Verified live 2026-08-15: device list `[RPC0, ROCm0]`, Vulkan0 never
+  listed. Consequence: the Strix half runs on the ROCm backend — better
+  prefill, ~3% slower decode than Vulkan0 (measured on this box).
+  A second head build (`/home/dank/llama.cpp/build-vulkan-rpc`, HIP off,
+  Vulkan on, RPC on) was built the same night for an A/B; with it the
+  device list becomes `[RPC0, Vulkan0]`.
+- **Default split is proportional to *reported free memory*, which is wrong
+  here.** The Strix reports up to 96 GiB (GTT ceiling) vs the 3090's 24 GB,
+  so the default lands ~20/80 in favor of the slow device (verified with a
+  4B smoke test: 4 of 36 layers on the 3090). We pin `--tensor-split 1,1`
+  (50/50) explicitly.
+- **`-dio` (direct I/O) on the head.** Large model + RPC load hangs without
+  it (visorcraft Strix finding; matches our 28 GiB load-stall history).
+- **Version skew hangs the handshake.** Worker built at the identical
+  commit (9b05354ec / b10433) in a worktree
+  `/home/dank/llama-cpp-rpc` on Dankserver — production b10121 build
+  untouched.
+
+Smoke test (2026-08-15, 4B model, 0.2/0.8 split, production untouched):
+worker VRAM rose ~819 MiB (real split), 95 t/s prefill, 58.8 t/s decode on
+the tiny model. RTT on the link: ~0.3–0.5 ms.
+
+Full design, scripts, and morning runbook:
+[Qwen3.8 PP Deployment](Pipeline%20Parallelism/Qwen3.8%20PP%20Deployment.md).
+
+### 2. Speculative-decoding tuning (same bits, free tokens)
+
+- **MTP `n_max`.** Production leaves `--spec-draft-n-max` unset. Our matrix
+  (Vulkan, 16k) found 13.3 t/s at the tested setting vs 7.2 without spec;
+  the published Strix tuning curve peaks at n_max 3–4 (30.3 t/s at 4 vs
+  28.9 at 3 vs 29.9 at 5). The PP env sets **4** explicitly; the matrix
+  re-tests 3/4/5 on the real split.
+- **Q4_0 draft head.** The published Strix card measured a standalone
+  Q4_0 MTP head ~6–10% faster than Q8_0 for the same acceptance (the draft
+  head is a 248k-vocab output projection; cheaper dequant, same accuracy).
+  The bundled head inside UD-Q8_K_XL is Q8-class; the standalone
+  `ggml-org/mtp-Qwen3.8-27B-Q4_0.gguf` (1.56 GB) is being downloaded for
+  an A/B via `--spec-draft-model`.
+- **Ngram-mod fallback stays.** It is the only spec type that survived
+  parallel load (DSpark crashed ROCm, DFlash corrupts multi-slot).
+
+### 3. Same-bit quant A/B: UD-Q8_K_XL vs plain Q8_0
+
+Both are 8-bit. UD (unsloth "dynamic") mixes block sizes per tensor; plain
+`Q8_0` uses uniform 32-element blocks. Dequant of a smaller block costs
+more ALUs per element; on a memory-bound GPU the difference is usually
+small, but Strix's iGPU has tight ALU headroom. `ggml-org/Qwen3.8-27B-GGUF`
+ships a Q8_0 (29.12 GB). One restart to test; if it wins, same quality at
+zero quality cost.
+
+### 4. Prefill is the agentic bottleneck — route it right
+
+Every agentic turn re-prefills the whole conversation (no prefix caching for
+hybrid state, upstream). Levers:
+
+- **Backend choice per workload.** Measured @64k on this box (b10433):
+  ROCm 410 t/s vs Vulkan 302 t/s prefill; Vulkan 25.1 vs ROCm 24.3 t/s
+  decode. For chat-heavy use: Vulkan. For agentic/batch-heavy use: ROCm.
+  The PP head (ROCm side) gets this for free.
+- **ubatch/threads-batch.** Production runs ubatch 256. Larger ubatches
+  (512/1024) fill the iGPU better during prefill; `--threads-batch` for
+  the CPU-bound tail. Untested at prefill scale on this box — matrix item.
+- **KV q4_0** for very long contexts: halves the 16-layer KV (8 GiB → 4 GiB
+  at 262k). Only worth it if long-ctx attention is the bottleneck; q8_0 is
+  the safe default.
+
+### 5. OS level (from the Improvements doc, re-verified 2026-08-15)
+
+- **TTM `pages_limit` is still 3,995,281 pages ≈ 15.24 GiB** — verified
+  today, unchanged. The 96 GiB GTT budget only materializes if the TTM cap
+  is raised (kernel parameter, needs reboot). Until then the effective
+  ceiling is lower and the 28 GiB load-stall risk remains. **Highest-value
+  single OS change; schedule with the next reboot.**
+- `iommu=pt` — verified present on the kernel cmdline. Done.
+- **Vulkan ICD pinning** — `radeon_icd.json` exists; no `VK_ICD_FILENAMES`
+  or loader-layer env is set. Pinning the radeon ICD and disabling loader
+  layers (`VK_LOADER_LAYERS_DISABLE=all`) is the recommended Strix
+  hardening; testable as service env vars, zero risk.
+- **Power** — GPU DPM is already `performance`, forced level `auto`. The
+  `power_profile` sysfs file assumed in the Improvements doc does not exist
+  on this machine (checked 2026-08-15); that item is N/A here.
+- Threads: 12 of 16 cores in production. Community reports 12–16 are
+  equivalent; leave at 12.
+
+### 6. llama.cpp version watch
+
+b10433 (2026-08-15) is adequate. Through b10451 there are no
+spec-decoding fixes that help us; DFlash's multi-slot corruption is still
+open upstream (fix #27117 pending). If that lands, DFlash/DSpark become
+re-testable candidates (DSpark was the best single-machine draft before the
+ROCm crash). No rebuild planned until then.
+
+## Quantization landscape (reference)
+
+The constraint is to keep UD-Q8_K_XL. This section records what exists so
+future decisions are informed.
+
+| Format | Size | Bits/weight | Notes |
+|--------|------|-------------|-------|
+| **UD-Q8_K_XL (unsloth)** | 31.5 GB | ~8.5 (mixed) | **Current.** Dynamic mixed-block; bundled MTP head; our baseline. |
+| Q8_0 (ggml-org, bartowski) | 29.1 GB | 8.0 | Uniform 32-block. Same-bit A/B candidate (lever 3). |
+| UD-Q6_K_XL (unsloth) | ~22 GB | ~6.5 | Below the 8-bit floor; documented, not recommended under the constraint. |
+| Q6_K | ~21.5 GB | 6.6 | Same tier as above. |
+| **ROCmFP4 "STRIX" (ROCmFPX fork)** | 13.3–13.8 GB | 4.25–4.39 | Strix-specific NVFP4-style format, ROCm backend only, requires a forked llama.cpp. Published: 30.3 t/s vs 10.7 for Q4_K_M (2.83×) at 8k with MTP n_max≈4; PPL ≈ Q4_K_M. **This is the "fast path" — and it is a quant downgrade plus a fork dependency that breaks the Vulkan production path.** Kept here as the escape hatch if the 8-bit speed target cannot be met. |
+| NVFP4 (Blackwell) | ~14 GB | 4.5 | Hardware path for RTX 50-series. Irrelevant to 3090/Strix. |
+
+If the 8-bit PP result (lever 1) is disappointing, the decision order is:
+push PP split/latency tuning → ROCmFP4 as a *second* service (A/B, not a
+replacement) → revisit the constraint with Frank.
+
+## Morning test matrix (2026-08-16)
+
+Policy: stop both production services, run tests, restore. Each config =
+one head restart (~10 min load incl. weight stream; ~2 min with the worker
+tensor cache warm).
+
+| # | Config | Measures |
+|---|--------|----------|
+| A | PP split 1,1, ROCm head, MTP n_max=4 + ngram | baseline PP |
+| B | A + `SPEC_NONE=1` | raw (no spec) PP speed |
+| C | A but head build `build-vulkan-rpc` (Vulkan side) | backend A/B |
+| D | A + `--spec-draft-model` Q4_0 MTP file | draft-head A/B |
+| E | A with n_max 3 and 5 | n_max curve on the split |
+| F | A, plain Q8_0 model file | same-bit A/B |
+| G | PP default proportional split (no `--tensor-split`) | confirms the 20/80 handicap |
+| Ref | production frank-default-large (Vulkan, 2 slots) | the bar to beat |
+
+Per config: `pp-test.sh bench 4` (4k prompt / 256 out) and `bench 16`
+(16k / 256). Report: decode t/s (server print_timing), prefill t/s, MTP
+acceptance, worker VRAM, head GTT usage.
+
+Success bar: **≥ 30 t/s at 4k and ≥ 20 t/s at 16k decode** (vs 25 t/s
+production @16k) with the same 8-bit model.
+
+## Sources
+
+- GGUF header dump of the production model (this machine, 2026-08-15).
+- Qwen3.8-27B model card + `ggml-org/Qwen3.8-27B-GGUF` /
+  `unsloth/Qwen3.8-27B-GGUF` repo listings (Hugging Face).
+- kingjones `Qwen3.8-27B-ROCmFP4-GGUF` Strix tuning card (MTP n_max curve,
+  Q4_0 vs Q8_0 draft head, ROCmFP4 numbers).
+- visorcraft `strix-halo-llm-perf` (Vulkan tg / ROCm prefill, ICD pinning,
+  `-dio` for RPC).
+- llama.cpp b10433 source: `src/llama.cpp` (device selection),
+  `common/speculative.cpp` (MTP/ngram-mod), `ggml/src/ggml-rpc` (protocol).
+- Local benchmarks: [frank-default-large-benchmark](frank-default-large-benchmark.md),
+  [qwen3.8-spec-matrix](qwen3.8-spec-matrix.md).