PP kit + Strix research: Qwen3.8-27B RPC split (Dankclaw head / Dankserver worker)

- scripts: serve-pp-head/worker, envs, systemd units, pp-test.sh orchestrator, pp-bench.py
- head design verified in b10433 source + live 4B smoke test:
  * no --device with --rpc (RPC device silently dropped otherwise)
  * default device selection = [RPC0, ROCm0] (HIP registers before Vulkan)
  * default split proportional to reported free mem = ~20/80 here, pin 1,1
  * -dio required for large model over RPC
- docs: qwen3.8-strix-optimizations (ranked no-quant-downgrade levers,
  quant landscape, 2026-08-16 test matrix), PP deployment runbook,
  index updates, Dankclaw Improvements corrections (Vulkan0, no power_profile sysfs)
- worker built at 9b05354ec in /home/dank/llama-cpp-rpc worktree (CUDA+RPC)
- second head build build-vulkan-rpc (HIP off) for backend A/B
This commit is contained in:
Frank committed 2026-08-16 05:40:27 -07:00
1 parent c775077016
commit 9544c36afc
13 files changed
+1037 -5

No files matched your search

+20 -4
View File
@@ -6,7 +6,23 @@ inference runtime tuning, and server/OS/hardware settings. Each item lists a
priority and the exact action.
Dates: researched 2026-08. Hardware is verified on the live machine unless
marked "verify first".
marked "verify first". Re-verified 2026-08-15/16; corrections marked below.
## Corrections (re-verified 2026-08-15/16)
- **Backend**: production is **Vulkan0**, not ROCm. Measured on this box
(b10433): Vulkan 25.13 t/s vs ROCm 24.33 t/s decode at 16k; at 262k ctx
Vulkan decodes 48% faster (5.68 vs 3.83 t/s). ROCm wins prefill (410 vs
302 t/s @64k). The "HIP out-performs Vulkan for decode" claim did not
hold on the 8060S. See [[frank-default-large-benchmark]].
- **Power profile**: the `/sys/class/drm/card0/device/power_profile` file
does not exist on this machine (checked 2026-08-15). GPU DPM is already
`performance`, forced level `auto`. This item is N/A.
- **KV cache**: production runs `q8_0` K **and** `q8_0` V (both quantized).
- **Spec**: DFlash/DSpark are retired. Production is `--spec-default`
(ngram-mod) with draft-mtp first. See [[qwen3.8-spec-matrix]].
- **TTM**: still 15.24 GiB as of 2026-08-16 — the modprobe fix is still
pending a reboot.
## Verified Memory Picture (measured 2026-08-11)
@@ -47,8 +63,8 @@ Verified against the running service and llama.cpp source.
| Change | Recommendation | Priority |
| ------ | -------------- | -------- |
| Backend | Keep HIP/ROCm. HIP out-performs Vulkan for decode on RDNA3.5/4 iGPU. | Done |
| KV cache | Keep `q8_0` K / `f16` V. q8_0 halves memory vs f16 with negligible loss. Try `q4_0`/`iq4_nl` only if more context is needed. | Keep |
| Backend | **Vulkan0** (2026-08-15, corrected — see Corrections). ROCm only for prefill-heavy workloads. | Done |
| KV cache | `q8_0` K / `q8_0` V in production. Try `q4_0` only if more context is needed. | Keep |
| Flash attention | Leave `--flash-attn auto`. It helps prompt processing, not decode. | Keep |
| Load mode | Start with mmap (default). Enable `--mlock` only if page-fault stutter appears. | Optional |
| Threads | Use `--threads 12 --threads-batch 16 --threads-http 1` for 16 Zen5 cores + 3 slots. Frees cores for the ROCm driver. | Medium |
@@ -62,7 +78,7 @@ All require root and a reboot for the modprobe change.
| Setting | Action | Effect | Priority |
| ------- | ------ | ------ | -------- |
| TTM GPU pool | Set `amd-ttm --set 96` (write `/etc/modprobe.d/ttm.conf`), reboot | Lifts the 15 GiB TTM cap toward full RAM; more headroom for GPU buffers | High |
| Power profile | `echo performance > /sys/class/drm/card0/device/power_profile` | Sustained inference speed; apply at boot via a udev/systemd unit | Medium |
| Power profile | **N/A on this machine** (2026-08-15): no `power_profile` sysfs; GPU DPM already `performance` | none | ~~Medium~~ |
| NVMe scheduler | Already `none` (`[none]` confirmed) | No change | Done |
| BIOS VGM | Set Variable Graphics Memory / UMA to "High" in the EVO-X2 BIOS | Max dedicated VRAM for the GPU carve-out; verify in the live BIOS | Verify |
| Firmware/EC | Check GMKtec for a newer EC/fan-curve firmware | Better sustained thermal behavior | Verify |